Journal of Computer Assisted Learning

1.1 Practising and student engagement

Despite some obvious challenges in keeping school‐aged adolescents engaged throughout the learning process, there are some clear indications regarding phenomenological, instructional, teacher, individual, and school factors on how to avoid potential alienation and disconnection (Shernoff, Csikszentmihalyi, Schneider, & Shernoff, 2014). Relevant for the process of practising content is that student engagement notably differs with respect to whether they like the task at hand or they consider it to be challenging and important. Following the principles of flow theory, students should simultaneously experience concentration, interest, and enjoyment when practising in order for a more complex set of subject‐specific skills to develop (Csikszentmihalyi, 1997).

Given that computerized content practising is often implemented as a low‐stakes activity, students should consider the task worth doing for its own sake. Expectancy‐Value Theory, first developed by Atkinson (1957), posits that an individual's choice, persistence, and performance can be explained by their beliefs about how well they will do on a certain activity and the extent to which they value the activity. Whereas some of the task value constructs are in ways similar to other theories of motivation, most notably self‐determination theory (Deci & Ryan, 1985; Ryan & Deci, 2000) they originate from different perspectives and are also distinct in terms of applications. Task value can be described in terms of its three subcomponents interest (i.e., intrinsic value), usefulness (i.e., utility value), and importance (i.e., attainment value), of which the latter two have been found to significantly predict effort and performance on low‐stakes tasks (Cole, Bergin, & Whittaker, 2008).

For adolescents working on academic tasks of considerable difficulty, self‐esteem maintenance motives are considered pivotal determinants of task value, which, in contrast to predictions for adults, is decreasing in perceived task difficulty (Eccles & Wigfield, 1995). Also, in determining the relative importance of expectancies for success and task value, the latter is considered particularly important in motivating students to engage in low‐stakes activities, which is often how computerized practising is implemented in the educational curriculum (Eccles & Wigfield, 2002; Wigfield & Eccles, 2000).

Personalizing the practising process would further promote continuous effort to practise as it better addresses individual learning needs, thereby increasing perceived expectancies for success and task value (Eccles et al., 1983; Tomlinson et al., 2003). On the other hand, personalization can have ambiguous, or even negative, effects on student engagement through, depending on its consequences with respect to task difficulty. Whereas increasing task difficulty generally improves task value for mature learners, this result is not evident for adolescents as they are prone to disengage in settings where tasks are perceived too difficult (Eccles & Wigfield, 1995).

Personalized practising can thus result in lower student engagement, even when the needs of students are better addressed. For instance, a personalized version of computerized practising may return more difficult exercises to students after they have performed relatively well on previous tasks. Due to this increased difficulty level, students may become frustrated when using the tool and actually become less engaged, even though practising with these more difficult exercises is better for them in improving their overall learning. It is therefore important to empirically establish the intricate relationships between task difficulty, computerized practising, and student performance. Lastly, a recent meta‐analysis on educational technology applications in K‐12 education indicates that improvements were markedly different for students of low and medium ability (Cheung & Slavin, 2012), suggesting that it is important to estimate results at different levels of (subject specific) student ability.

1.2 Two approaches towards computerized practising

Table 1 characterizes features of both versions of computerized practising considered in this study. The table shows that both versions recognize that exercises can have different difficulty levels and address different knowledge types. Each exercise that can be offered to the student is therefore labelled with a knowledge type and difficulty level. The knowledge type level indicates the level of learning complexity, based on the widely adopted and revised version of Bloom's taxonomy (Anderson et al., 2001; Bloom et al., 1956; Krathwohl, 2002). The levels considered here are (1) memorization (replication), (2) active use of knowledge (application), and (3) comprehensive understanding (insight).

The difficulty level for each exercise is determined on the success rate of, on average, 3,000 students from previous cohorts that have worked with the tool. Based on this historical success rate, each exercise is assigned to one of three difficulty levels. The first interval represents relatively easy exercises (33.33% highest success rates), the second interval to levels of moderate difficulty (between 33.33% and 66.67% highest success rates), and the third interval are exercises for which the success rate is among the lowest 33.33%. Both versions also provide feedback while practising, which has been found to be one of the most important factors in facilitating learning (Hattie & Timperley, 2007). A student receives feedback immediately after answering an exercise on whether the answer was correct and with cues on where to find information relevant to improve their level of understanding for that particular exercise. Even though each student receives the same type of feedback, it could be that the given feedback to students in the personalized practising version is more effective because it better matches the current knowledge and performance level of the student (Shute, 2008).

One important difference between both versions is also how difficulty levels are incorporated in the practising process. The non‐personalized version distinguishes between these three difficulty interval levels and the aforementioned three knowledge types. Each practising session randomly selects a string of exercises for each topic; orders these exercises subsequently by topic, knowledge type, and difficulty level; and returns this string of exercises in this specific order to the student. It follows that the non‐personalized practising condition simulates a more traditional practising approach in which the path of returned exercises is predetermined in terms of topics, knowledge types, and difficulty levels.

The personalized practising condition, however, does not generate a predetermined string of exercises but instead returns exercises contingent on the topic‐specific real‐time practising performance of the individual student. The practising process is personalized by conditioning on (a) the current individual topic‐related understanding level relative to that of the class, (b) mastery learning, and (c) effort sustainment. The individual understanding level reflects the proportion of correct answers on a certain topic. The personalized version considers relative understanding levels because individual understanding levels may reflect the general difficulty level of a particular topic, rather than the poor or below average understanding of the student. The programme updates these relative understanding levels every time a new exercise is assigned. The probability that a particular topic is returned is inversely proportional to these relative understanding levels. In other words, the more difficult a topic is for an individual student, the more likely (s)he will receive an exercise on this topic.

Mastery learning within the context of personalized practising requires that students can only advance to the next knowledge type level when the current level is sufficiently mastered. It has been shown that mastery learning positively influences student achievement (Kulik, Kulik, & Bangert‐Drowns, 1990; Slavin, 1987) and positively affects student attitudes towards content and instruction (Kulik et al., 1990). The latter finding suggests that the personalized practising approach has the potential to improve student engagement. The personalized version incorporates this by returning a question of a particular knowledge type before a threshold level of success rate is obtained.

Effort sustainment is an important prerequisite for practising to promote learning. Following flow theory, keeping students engaged requires practice material to be just a bit in advance of the student's current level of readiness (Csikszentmihalyi, 1997). The personalized version addresses this by offering exercises of a success rate that is slightly above a student's current level of readiness. If the exercise turns out to be too difficult to complete successfully independently, the assumption is that the task will still be in the zone of proximal development (Vygotsky, 1978) and that the feedback provided by the software provides the student with the necessary scaffolding and support. Studies confirm that students are indeed more likely to sustain efforts when working at tasks at moderate levels of difficulty, compared to when tasks are either too difficult or under challenging (see Tomlinson et al., 2003, for a review). When problems are consistently too difficult, low success rates can refrain students from further practising. For example, empirical research on math anxiety indicate that frustration with practising can be an important mechanism in explaining poor performance (Ashcraft, Krause, Hopko, Berch, & Mazzocco, 2007). At the same time, when tasks are consistently too easy, students, despite high success rates, are likely to lose motivation and also avoid further practice (National Research Council, 2003).

Practice intensity may thus decrease substantially when exercises offered are either perceived to be too easy (i.e., “boring”) or too difficult (i.e., “frustrating”), which is more likely to occur in the non‐personalized version of practising. It is thus relevant to empirically address how practice intensity develops over time and to distinguish between assignment conditions. A detailed technical discussion of the algorithms underlying both computerized practising versions is given in Section 2.3.

2.1 Dutch education system and participants

The general structure of the Dutch educational system is shown in Figure 1. Dutch compulsory schooling law requires children to start with primary education in the year they turn 5 years old. Children are tracked into different secondary education levels upon finishing primary education (Grade 6). This decision is based on the test scores achieved on a standardized nationwide test and on the advice of the primary school teacher. Children can be tracked into three secondary education levels: (1) prevocational education (4 years), (2) general secondary education (5 years), and (3) preuniversity education (6 years). Prevocational education prepares children for vocational education, secondary general education prepares children for universities of applied sciences, and preuniversity education prepares children for universities. This research focuses on students enrolled in preuniversity, upper secondary general education, and prevocational secondary education. All student are enrolled in Grades 7, 8, or 9.

Five secondary schools participated in this experiment with in total 28 different classes. All 654 students were randomly assigned within classes to a personalized and a non‐personalized version of the digital practising tool, such that the effects of the personalized practising condition on task perceptions and student performance can be causally evaluated. The practice intensity and performance of these 654 students was tracked from August 2014 to June 2015, in which over 150,000 exercises were answered in the digital practising environment. Of these students, 618 (94%) filled out the pre‐experiment questionnaire regarding background characteristics. Four hundred fifty‐five students (70%) also took part in the postexperiment questionnaire, asking them about their experiences with using the tool. A description of the participants is shown in Table 2. The comparison with students who did not complete the final survey suggests that there are some patterns of selective nonresponse, in that respondents are slightly older, less likely to have a parent born in the Netherlands, perform better on the prescore, and more likely to have higher educated parents. The comparison between students in the personalized and non‐personalized version supports that randomization was largely successful in generating two groups equal in expectation. A notable exception is the proportion of parents with education level “Middle,” which is higher for students working with the personalized version. However, this does not extrapolate into differences in the more extreme levels of education (i.e., “Low” and “High”), thereby yielding inconclusive support for any structural differences in parental education between the two experimental groups. Successful random assignment is further corroborated by the similarity in prescores, defined as the last test score for that subject prior to when the class started using the digital practising module. Importantly, whereas there has been some attrition with respect to participation in the postexperiment questionnaire, this does not seem to be related to the experimental condition students were assigned to. This makes it possible to draw causal inferences when comparing these two versions of the digital practising tool.

2.2 Materials

Figure 2 visualizes the practising environment of students (screen on the right) and the dashboard used by teachers to monitor the practising behaviour of their students (screen on the left). Students log in with their personal access code and indicate which subject and chapter to practise with. They are then redirected to a screen similar to the one shown in Figure 2. The top bar shows the chosen subject and chapter, and below this bar information is displayed on the students practice performance (determined by practice duration and performance) and the number of exercises already completed. The practising environments of the personalized and non‐personalized condition are identical.

The dashboard for teachers displays information on the practice activity of the class and provides extensive information on the practising process of the individual students (i.e., practice time, number of exercises answered, and proportion exercises answered correctly [by knowledge type]). Teachers are free to choose to use this information when providing feedback to their students about their learning process.

Throughout the year, each practice session for a student is linked to the corresponding test for which the student is preparing. In general, students have a few weeks of classes, with corresponding homework, before a test is administered. After this, the next period starts, in which a new practice session is generated for each student in each of the participating subjects and with exercises linked to the next test for that particular subject. For each session, the student can voluntarily decide to take up the opportunity to participate or not (i.e., participation decision). During the year of the experiment, four subjects participate in the experimental sample: Dutch, Biology, History, and Economics. For each session, it is monitored whether or not the student decides to practise (i.e., participation decision), together with the number of exercises, the success rate (i.e., proportion correct), total practice time (i.e., in minutes), and performance on the corresponding summative test (i.e., scale of 1–10).

At the end of the school year, all students engaged in a postquestionnaire, asking them about their experiences with, and perceptions towards, using the computerized practising software. This postexperiment questionnaire asked students to evaluate statements concerning whether the tasks were generally “too easy” (TE: “When using the tool, exercises I receive are often too easy”), “too difficult” (TD: “When using the tool, exercises I receive are often too difficult”), “interesting” (TV_i: “When using the tool, exercises I receive are interesting.”), and “useful” (TV_u: “The tool is useful in helping me to better understand the curriculum”). Ratings were performed on a 5‐point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree). Answer 3 defines the neutral reference category. Answers 1 and 2 are taken together, as well as 4 and 5, as to define whether a student expresses a particular disposition (positive or negative) towards computerized practising. This classification is used to empirically compare the group of students expressing a disposition towards the task to those responding neutral to the statement (i.e., reference group). An overview of the items and coding in the survey is provided in the Table 3.

The aforementioned task perceptions will be used in the analyses to gauge the extent to which the practising process is likely to promote or inhibit student engagement. Differences in these perceptions will be related empirically to two outcomes indicative of student engagement. First, every time a new practice period starts (i.e., in between two summative tests), a student will make the decision whether or not to take up the opportunity to voluntarily practise using the software. This participation decision will be influenced by whether previous experiences with the tool were generally positive or negative and a higher likelihood of participating in computerized practising is thus considered to be a sign of greater student engagement. Second, conditional on deciding to participate, students will use the tool longer (shorter), depending on whether they are more (less) engaged in the process. As such, a higher level of practice intensity (measured in minutes) in a particular session is considered to also be a sign of greater student engagement.

2.3 Practising algorithms

The two versions of computerized practising considered in this study are different in terms of their underlying algorithms. Below these technical differences are described in more detail.

2.3.1 Non‐personalized practising

The algorithm underlying the non‐personalized practising condition is intuitively illustrated in Figure 3. Each exercise, e, received a label which refers to the topic (t), the knowledge type level (k), and the level of difficulty (d). It follows that e_n^t, k, d refers to exercise n in exercise subset {t, d, k}. The non‐personalized version distinguishes between three difficulty interval levels and three knowledge types, which explains why Figure 3 shows nine exercise subsets per topic.

For each student, one exercise is randomly drawn from each subset {t, d, k}, beginning with e^1, 1, 1 (the upper‐left red coloured square) and ending with e^T, 3, 3 (the lower‐right red coloured square). The selected exercises are ordered by, subsequently, topic, knowledge type, and difficulty level and returned one by one to the student in this specific order. Exercises are randomly selected and are an integrated part of the learning material that is offered in the school, such that unique practising sessions are generated for each student that are valid and representative for the upcoming summative tests.

Some subsets contain more exercises than other subsets, which are represented by the height (h) of the square around e^t, k, d in Figure 3. The probability that a particular exercise is returned from subset e^t, k, d equals /(). This illustrates the importance of having enough content, because the probability that the algorithm offers a particular exercise to a student becomes larger as the amount of content decreases.

2.3.2 Personalized practising

The algorithm of the personalized practising tool is visualized in Figure 4. The understanding level of student i and class c on topic t is determined by the proportion of correct answers given. Let P denote this proportion for topic t, then the relative understanding level of topic t for student i is . Higher values of _t refer to better understanding levels of student i. Figure 4 shows that the width of each box (l_t) represents the inverse relative understanding for student i of a particular topic and the total total length l is an indicator for the inverse of a student's overall relative understanding. It is convenient to take inverse values because larger l_t values indicate that topic t is less well understood and, as such, represents the probability that the algorithm will offer an exercise on topic t. The personalized practising version updates the relative understanding levels after each exercise practised and the probability that student i receives an exercise on topic t equals

(1)

After the algorithm has selected a topic t, it determines two parameters (μ and p). Parameter μ represents a topic‐specific threshold level which determines the knowledge type of the exercise that should be returned. This threshold level is set at 50%, which means that exercises of the current knowledge type are offered as long as does not exceed 0.5. The algorithm switches and offers exercises of the next knowledge type if and only if  > 0.5. Once the topic and the knowledge type are determined for a particular student, the algorithm returns an exercise for which holds that the understanding level of the average student is closest to the understanding level of the student (from above). In other words, an exercise will be returned that is slightly more difficult than the student's current understanding level.

The height of the square around e^t again refers to the available number of exercises on topic t, and the availability of sufficient content is of vital importance for the proper functioning of the personalization process. The figure illustrates a situation in which there are more exercises available for Topic 1 than for topic T . At the same time, it indicates that the student's relative understanding of topic T is smaller than that of Topic 1 (l_T  <  l₁), which means that the algorithm will return an exercises of topic T with higher probability. The consequence of not having enough practising material would be that identical exercises are returned to the student frequently, such that the practising process is not so much personalized, but rather a process in which wrongly answered exercises are repeatedly returned. Furthermore, limited content would also make it difficult for the algorithm to closely match a student's individual level of relative understanding with an exercise to return. In this study, each subset contains many exercises for the algorithm to choose from (see also Footnote 1).

3.1 Perceived task easiness and practice intensity

Students were asked to indicate if they generally experienced computerized practising as too easy [TE(−)], neutral [TE(0)], or as not too easy [TE(+)]. The relationship between perceived task easiness and practice intensity (measured as average practice time per session) is shown in Figure 5. In both conditions, there appears to be no clear difference between students who perceive computerized practising as either too easy, neutral, or not too easy.

The results reported in Table 5 Model 1 suggest that there is no clear relationship between perceived task easiness and the decision to participate in a particular practice session. For students who did practise, results for whether perceived task easiness affects practice intensity (Model 2) and student performance (Model 3) also indicate that perceived task easiness is not related to these outcomes either.

3.2 Perceived task difficulty and practice intensity

Students were also asked to indicate if they generally experienced computerized practising as too difficult [TD(−)], neutral, or as not too difficult [TD(+)]. The relationship between perceived task difficulty and practice intensity (measured as average practice time per session) is shown in Figure 6. In contrast to “task easiness,” the notion of “task difficulty” is systematically related to practice intensity. For students being assigned to the non‐personalized practising conditions, it holds that practice intensity is positively related to not perceiving the task to be too difficult. The practice intensity for students assigned to the personalized practising is also lower if they perceive computerized practising as too difficult. However, in this condition, there appears to be no difference between students who perceive computerized practising as neutral and not too difficult. A potential explanation for this is that students assigned to personalized practising receive on average more difficult exercises (see Table 4). As a result, fewer students stated that practising sessions were perceived as not too difficult, as illustrated by the relatively large confidence intervals for this group. Those students experiencing practising as not too difficult receive on average more difficult exercises and this could cause convergence to the level of practice intensity of students who were neutral with respect to the difficulty level of computerized personalized practising.

The estimation results in Table 6 Model 1 suggest that there is no relationship between perceived task difficulty and participation in practising. For students who did participate, the estimation results in Model 2 suggest that practice intensity is reduced when the perceived task difficulty is too high and increased when the task is considered not too difficult. However, only when task difficulty is considered too high, do the results appear to be statistically significantly different from the neutral category. Furthermore, the perception that the task of computerized practising is too difficult (or not), does not contribute to achieving higher (or lower) test scores (Model 3).

In comparing the results for task easiness and task difficulty, it can be concluded that only the perception that the task is too difficult seems to be relevant in explaining differences in student engagement with computerized practising, as indicated by observed lower levels of practice intensity.

3.3 Task value and practice intensity

To infer elements of task value, students were asked to report whether they perceive computerized practising as interesting [TV_i(+)] or useful [TV_u(+)]. Figure 7 shows the relationship between practice intensity and the extent to which students perceive computerized practising as interesting. In general, the figure illustrates that practising intensity increases when students perceive practising as more interesting. Somewhat remarkable is that for students assigned to the non‐personalized practising condition the practising intensity of students who consider this task to be interesting is not higher than that of students who find practising less interesting. The confidence intervals reveal that these differences are not statistically significant and, furthermore, these differences may be driven by differences in participation decisions and background characteristics.

Table 7 Model 1 shows that students who perceive practising as interesting are indeed more likely to participate in the practising process, but only if these students were assigned to the personalized practising process. Students in the personalized practising conditions receive exercises that better match their personal needs throughout the entire practising process, and this may induce them to practise. Students in the non‐personalized practising condition may experience that the offered exercises do not match their personal practising needs and, as such, may be more likely to decide not to practise in upcoming sessions, even though they do find computerized practising interesting. Table 7 furthermore shows that perceiving computerized practising as interesting is not related to practising intensity (Model 2), nor test scores (Model 3), once students have decided to participate in the practising process.

Figure 8 visualizes the relationship between practice intensity and whether students perceive computerized practising to be useful. The results are very similar to those in Figure 7, in the sense that practising intensity increases when students perceive practising as useful, but only in the personalized practising condition. The estimation results in Table 8 Model 1 show that the participation decision of students to practise is not driven by their perceived judgment on whether practising is useful. Practice intensity of students who perceive the practising process as useful is significantly lower for students assigned to the non‐personalized practising condition, but higher for students assigned to the personalized practising condition (Model 2). Again, these results do not translate into differential patterns in test scores (Model 3).

In short, the results for task value interest and usefulness point out that both have the potential to promote student engagement, albeit in different ways and only when students work in a personalized practising environment. Students who perceive the task as interesting are more likely to take up the opportunity to practise when assigned to the personalized condition, but conditional on this effect, no differences in practice intensity are observed. For students who perceive the task as useful, the potential to improve student engagement in the personalized condition is not so much driven by a higher likelihood to participate, but by the fact that, once students decide to practise, they will do so for a longer period of time.

3.4 Practice intensity and task perceptions evolving over time

In the previous analyses, average practising intensities were related to the task perceptions measured at the end of the school year. However, the observed patterns of practice intensity over the school year is dynamic and, as a result, differential patterns in practising time may emerge over the course of the school year and across different levels of perceptions. This gives insights into the potential for student engagement to either improve or deteriorate as experience with the task increases. Figures 9 shows these patterns for the various task perceptions considered throughout this study.

As to be expected, early on in the year (i.e., in the absence of experience with the task), practice intensity is still largely similar for students who end up differing considerably in terms of task perceptions. Practice intensity initially increases with test number, which indicates that students tend to practise more over time throughout the school year after which it levels off. There are some notable differences in patterns between students belonging to different categories of task perceptions. In general, the observed increase in practice intensity is smallest for students who perceived practising as too difficult or too easy, which is in line with theoretical predictions. Furthermore, the differential results with respect to task value and assigned condition presented in the previous section are confirmed in these dynamic patterns and seem to emerge in the second half of the year (i.e., after test Number 4).

3.5 Effects of personalization on task perceptions by ability

Given that implications of technological interventions can differ importantly with respect to ability, the previous results are separated by prescore. This prescore is obtained from the last test prior to when the class started using the digital practising module for that particular subject and can be used as a proxy for subject‐specific ability. Several differences emerge between the two assignment conditions, as presented in Figure 10. The non‐personalized version is considered to be too difficult particularly by students with a relative low prescore. The personalized version is designed to adapt and expose better students to more difficult exercises and weaker students to less difficult exercises. This is confirmed in that there is no statistically significant difference in test scores between students who consider the exercises to be (not) too difficult in the personalized condition. With respect to task value, a general pattern is observed for both conditions in that students who find the tool interesting are generally students of lower ability (cf. Cheung & Slavin, 2013). However, a notable difference emerges with respect to whether exercises are considered to be useful. In the non‐personalized version, students evaluating the tool as not useful have somewhat lower prescores, which is even more the case for students considering the tool to be useful. Instead, for students assigned to the personalized version, it is particularly the relatively high‐performing students who evaluate the tool as useful. In short, the non‐personalized version particularly appealed as useful to relatively low‐ability students, whereas the reverse is found for the personalized condition.