Journal of Computer Assisted Learning

2.1 The experiment

The intervention in this study is that treatment students practice with the spelling modules of the computerized adaptive practice homework tool. The intervention is meant to practice the specific automatization skill for spelling, but this is a skill that can be more generalized as it is crucial to other aspects of literacy as well. Control students also practice with the online tool and are exposed to the tool for the same average number of minutes each week, but their accounts are technically disabled for using the spelling modules. They have access to the vocabulary modules instead. Teachers would not know which students practiced with which exact modules, so that they will treat all students the same, and no different treatment is to be expected from parents either, as all students had to practice with the online tool during the whole school year, the only difference was that topics practiced were different for treatment and control group in the first semester, which was reversed in the second. Besides the intervention, all students have regular language classes in which all aspects of language are being taught. All students have access to all other modules of language, such as text comprehension and grammar. A pilot study with the same program gave the indication that the online practice tool would possibly only be effective for modules that are easier to automatize and are not so much related to the method used and pace at school. Therefore, the modules of spelling and vocabulary were chosen. We originally planned to also study effect of practicing with vocabulary. However, there was such little variation in the outcome measures that these analyses would not have been reliable.

2.2 Characteristics of program under study

The purpose of the computerized adaptive practice tool is to help students practice their language skills, while being able to individualize, and give users direct feedback (Muiswerk, 2013). Although the program is mainly being used in the Netherlands, it also has an international version and is used by several international schools both in Europe and other parts of the world. The program is interactive and person specific. Students work at their own level and get those exercises that will help them improve the subaspects of language they are not knowledgeable in yet, whereas some exercises are meant to keep up their already acquired knowledge. The school uses this tool to make sure each student achieves the highest possible level of language, given his/her abilities, and maintains the level achieved. It offers all students online access to the tool for use after school hours, at home. In this sense, it is a supplemental program in the categorization of Cheung and Slavin (2012).

First year secondary students (seventh grade) make a language skills pretest at the start of the school year, in September. This test determines their level of different subaspects of language, which in turn determines the types of exercises they have to start practicing with at home. Note that a student questionnaire showed that all students have a computer or device at home that they can use for practicing. At regular intervals, students make a short computer test at school to determine for which exercises their skills are still lacking and for which exercises their knowledge level is good enough for the moment. After every test, the type and level of exercises a student can access are adjusted to their new skill level.

The program functions in a highly individualized manner, as it starts with explanation screens (digital instruction), offers feedback, and provides the student with either repetition or new learning modules on the basis of previous performance of this specific student. It works without teacher interventions, but teachers have access to a reporting module, and some may incorporate knowledge of performance in the computerized instruction environment in their interaction with the students.

Language teachers are supposed to motivate students to practice with the computerized instruction tool at home and to check whether students indeed practice the suggested number of minutes. However, not all language teachers are similarly convinced by the usefulness of the tool, and hence, teachers show different behaviour in motivating and checking up on their students. On the one hand, only few of the teachers actually log into the system to check their students' practice behaviour, but on the other hand, almost all teachers regularly motivate their students to practice the suggested 30 min per week. The program intensity is therefore low, in the categorization of Cheung and Slavin (2012). Despite the instruction of 30 min per week, of which 15 min with spelling (or vocabulary for control students), practice behaviour differs largely among students, as there is no sanction or test they fail if they do not practice.

The language skills are measured using digital standardized tests, which are written by all seventh grade students at T0 and T1 (see Figure 1). These are standardized validated tests, and these tests are based on other nationally validated tests. The reliability (Cronbach's alpha scores of between .79 and .92) and validity of these tests are analysed yearly by the tool developer, based on norm data of several participating schools (Schijf & Schijf, 2014). The tests are external to the practice exercise tool and do not contain any of the exercise questions. The tests measure whether students have mastered the required national language level they are supposed to have, given their age and given the fact that they finished primary school (called “reference level”). Note that this school uses quite some digital material and digital tests for other courses as well, implying that control group students also had a lot of exposure to digital teaching materials and are therefore not disadvantaged in this respect.

All students, both treatment and control groups, practiced with digital multiple‐choice language assignments in the testing program in the week before the pretest (T0) was administered, to make sure they knew what to expect when writing the pretest and to get acquainted with the testing environment. The test contains multiple choice questions. Test scores can range from 0 to 100, 100 being the absolute maximum.

2.3 Setup of the experiment

Figure 1 shows the timeline of the field experiment, which consists of a pre‐experiment period and the experiment itself. In spring of their final year in primary education, students register at their school of choice for secondary education. The secondary school used the results of the standardized national exit exam and the recommendation made by the primary school teacher to assign students to the first year classes before the summer break of 2013. At the school under study, the assignment of students to classes was done randomly within the boundary of the ability grouping that forms part of the Dutch system of secondary education (“early tracking”). Assignment of teachers to these new first year classes is fairly random as well, given that they can teach a certain number of classes each year, and it has to fit their (part‐time) schedule. In summer, Week 33/2013, the researcher assigned students randomly to treatment and control groups, taking into account the class of the student. In Week 34/2013, the school year started, and parents were informed about the experiment. Because control students would also practice their language skills, but with a different topic, and the assignment to exercises would be reversed after the experiment (in the middle of the school year), such that all students could practice with both types of exercises during the school year, all parents agreed to their child's participation in the experiment.

The pretest took place in Week 38/2013. The experiment lasted 16 weeks, excluding holiday breaks, and the posttest test (T1) took place in Week 5 of 2014.

3.1 Data

The school under study, Dendron College, has about 2,000 students in total and is—to Dutch standards—a mid‐sized school for secondary education (junior high and high school). Dendron College offers secondary education in all tracks and is tracking students from the first year on in several prevocational, general higher, and preuniversity tracks. Dutch secondary education has a tracking system from seventh grade on, with three different tracks (prevocational education, which consists of four subtracks where Level 1 is the lowest [mainly practical] track and Level 4 the highest [mainly theoretical] track, general higher education, and preuniversity education).

In school year 2013/2014, there were 377 students in 14 first year classes (equivalent to seventh grade in the US), ranging from the lowest prevocational track to the preuniversity track. First, 14 students were excluded from the analysis, as they were ill during either the pre‐ or the posttest, and therefore did not have test scores for both tests. Next, we lose 13 students, because these students do not have a primary school ability test score, the measure that is our second most important control variable after the pretest, and that we use to study heterogeneous effects. This leaves us with 350 students.

The age of the students in the experiment ranges from 11 to 13 (differences are mainly due to grade repetition), and about 55% is girls. The average score on the primary school ability test is 536, where the minimum is 509 and the maximum is 550. Note that the scores on this test have a theoretical range from 500 to 550 and that not all students have a score on this test, as primary schools can decide whether they use it or not. However, almost 90% of primary schools issues this specific test. The 14 classes have nine different teachers for language. Furthermore, students have very diverse backgrounds, as they attended 27 different primary schools. As such the school is a typical representative of schools outside of the highly urbanized, central region of the Netherlands (the “Randstad”).

3.2 Identification strategy

The main problem with determining the effect of a practicing tool is the potential correlation of unobservable factors with both the practicing behaviour and the outcome variables. In this study, we use exogenous variation in the possibility to practice through an experimental setup. Students were randomized at the individual level with a random number generator where even numbers where assigned to the treatment group and odd numbers to the control group, using a stratification strategy in which class is taken into account, to ensure an approximately evenly distribution of treatment and control group within a class. The latter is done to ensure that we have a good balance of lower and higher performing students in both treatment and control groups, because the Dutch tracking system places students of similar ability levels in similar classes. Note that we did not impose an exact 50–50 division over treatment and control groups and that some students were removed from the dataset, which in the end resulted in 185 treatment students and 165 control students.

First, we check whether there is balance between treatment and control students on observable characteristics. Although balance does not guarantee that members of both groups had equal expectations regarding the development of their skills at the start of the intervention period, it can at least give a good idea whether the randomization was successful. Using t‐statistics for continuous variables and dummies and chi‐squared statistics for the comparison of the nominal variables, we found that there is almost perfect balance between the two groups, with no significant differences in the mean values of the observable characteristics. A joint F‐test on all the characteristics also does not show a significant difference (F(66, 283) = 0.73, p = .94). Given these results, we expect the random assignment process to have functioned well, producing an intervention and control groups that were “equal in expectation” at the start of the intervention period.

3.3 Compliance with assignment

As it happens, not all students complied with their assignment in the experiment. Although we technically disabled the type of exercise a student was not supposed to practice with, students can still decide not to practice at all with the exercise they were supposed to practice with, and some did. This means that we have three groups of students: (a) students that were assigned to the treatment group (A = 1) and also practiced with this type of exercise in the online tool (P = 1), (b) students that were assigned to the control group (A = 0) and therefore did not practice with this type of exercise in the online tool (P = 0), and (c) students that were assigned to the treatment group (A = 1), but who did not practice at all with this type of exercise (P = 0). Comparison of these three groups of students shows that compliance is not a random process. The students that did not comply are “smarter” students (higher primary school ability test score and higher pretest score), mainly boys and often the youngest child at home. Possible explanations could be that smarter children have not felt the need to do their homework properly, as they get good grades anyway and that girls listen better to they teacher if they are being told that they have to practice online as homework.

Because we are interested in the effect of actually using the tool, this selective noncompliance is likely to create (a small) bias in the estimated effect. That is, a simple comparison of the control and experiment groups reveals the intent‐to‐treat (ITT) effect (showing what the effect is of offering the tool) but not the average treatment effect (showing what the effect is of actually using the tool), because a specific group seems to have self‐selected away from treatment. Therefore, we control for the noncompliance using an instrumental variable approach, as will be explained in more detail in Section 4. However, we also acknowledge that the problem of noncompliance is relatively small, so the ITT in itself is also already very informative.

3.4 Practice behaviour

As described above, the setup of the experiment is such that students are relatively free in deciding when, where, with what, and how much to practice, contrary to most other (relatively comparable) studies in the literature (Cheung & Slavin, 2012). Therefore, in addition to studying the effect of access to the digital practice tool, it is also interesting to look at the intensity of the treatment. Apart from that, teachers are mixed in their enthusiasm of the tool, in part because the freedom it offers to students makes it difficult to coordinate home practice learning with a regular, class‐based, and fairly homogeneous instruction process. Another reason is that they simply do not like the tool and therefore refuse to remind their students about practicing. However, remember that the tool is not used in class, but as homework, and that students are also reminded to do this homework via their digital calendar, suggesting that the teacher only has a limited influence on the intervention. Nonetheless, different enthusiasm by teachers also leads to differences in usage of the tool by students and also underlines the importance to control for the teacher in our analyses.

Below, we use the number of minutes practiced per week, which is determined by subtracting the weeks of school holidays from the total number of regular school weeks during the experimental period (i.e., excluding test weeks, leaving 16 weeks). We decided to subtract school holidays because students usually do not make homework in these weeks and would most likely also not practice with the tool. However, this means just rescaling, which will not influence the results in itself. Minutes practiced is registered by the digital practice tool for each time the student logs into the system and for each exercise the student practices with. In principle, students could also practice at home during school holidays, but the data show that this was hardly the case.

Figure 2 and Table 1 show the descriptive statistics of the practice behaviour of students. Figure 2 shows the spread of number of times practiced and the practice minutes over the weeks, and Table 1 shows the number of students who practiced, the total minutes practiced, the number of times students practiced on average, and the average practice minutes per week over the full 16 weeks. Figure 2 shows that the average time dedicated to spelling practice is fairly similar across weeks. It hovers between 5 and 10 min per week, with the exception of Week 5, where nine students practiced more than 20 min per week on average. Furthermore, it can be seen that between 40 and 100 students practiced each week. Although overall almost all students practiced at least once, this figure shows that in the best weeks (Weeks 11–14, when teachers were specifically reminded again to remind their students to practice in the tool), only about a 100 out of about 180 students actually practiced. Note that the fifth week was the autumn break, and Weeks 15 and 16 were Christmas break. So nine students were very active in their holidays, but as most students were not active, these weeks were not taken into account in the calculation for average minutes per week. The between student variation of practice behaviour across weeks probably also causes the difference between the average number of minutes practiced per week over the total period. Some students practice a very high number of minutes in 1 week but do not practice each week, whereas the others practice each week but only for a low number of minutes. Note that we do not see any differences in average practice behaviour between boys and girls and low‐, middle‐, and high‐performing students.

As said before, there are (very) large differences in practice behaviour between students. The average amount of time students practiced was 7 min per week for spelling, ranging from an average of 1 to 35 min per week. Note that practicing time is only counted if at least one exercise is finished in the episode. Over the total research period, a student practiced on average about 114 min for spelling, distributed over 40 different episodes, so on average a little more than twice per week. A questionnaire among seventh grade students in an earlier study on the same program showed that students write on average 15 to 45 min of Dutch homework per week. Therefore, an additional 7 min is an increase of Dutch homework time of 15% to almost 50%. So although the 7 min is only (less than) half of the supposed practice, it is still a considerable increase in language homework time for students.

5.1 Baseline results

The first results we present are the simple t‐statistics of the effect of access to the tool on multiple outcome measures, for spelling, in Table 2. Note that all coefficients in Table 2 are standardized coefficients, that are to be interpreted in terms of standard deviations with a mean of zero and a standard deviation of one (calculated based on the total group of students). First, we present the test score at T0, where we expect to see no significant difference, given the random assignment of the groups. Next, we present the absolute test score at T1, where it can be seen that there is a difference in T1 score between the treatment and the control groups, significant at the 10% level. We do not see a significant difference in the growth between T0 and T1. If we split the sample (approximately equally) by ability, we see that for the lowest performing group, we find a significant difference at the 5% level at T1 and a significant difference at the 1% level for the growth between T0 and T1. We do not see any significant results for the middle‐ and higher performing groups. But, as discussed before, not all students complied with their assignment to the treatment group, for both domains, and not all students practiced the same amount of time, and therefore, we should explicitly control for this if we would like to analyse the effect of actually using the tool, instead of only having access to the tool.

5.2 The effect of access to and usage of the digital practice tool

Table 3 presents the results of (a) the first stage, in which we analyse whether the randomization into the treatment group indeed highly predicts whether students use the tool, (b) the intent to treat, in which we analyse what the effect is of offering the tool to students, (c) the ordinary least squares (OLS), which is the naïve regression in which we analyse the effect of usage of the tool without controlling for the fact that students select themselves into using the tool (as discussed above, this is not a random process), and (d) the 2SLS analyses where we analyse the effect of using the tool while at the same time correction of this selection of usage.

Note that all coefficients in Table 3 are also standardized coefficients with respect to the standard deviation, similar as before. Table 3 shows that the first stage is highly significant, with a high r‐squared, and that the assignment to the treatment of the experiment significantly and highly influences the chance to practice with spelling in the digital tool. In the second part of Table 3 (top right), we see the ITT analysis, which does not take into account the actual usage of the tool but looks at offering the tool. This analysis shows that, once controlled for ability by including both the pretest and the primary school ability test, there is a positive significant effect of assignment to the experiment on the spelling posttest score. Students with access to the tool score on average .14 of a standard deviation higher than students that did not have access. In the bottom part of Table 3, we see the OLS and the 2SLS results, where we analyse the effect of actual participation in the experiment on spelling posttest scores. Note that the results of the second (top right; third and fourth analyses [bottom left and right, respectively; ITT, OLS, and 2SLS]) are not significantly different from each other. The OLS analysis (bottom left) shows that students that practiced with the tool score on average .15 of a standard deviation higher on the posttest than students that did not practice. But, as we saw before, this is not a random group of students, and in the OLS analysis, we do not control for that. Therefore, we turn to the 2SLS analysis (bottom right), where we use the assignment to treatment as an instrument for actual participation in the experiment to control for this selection issue (in a way, we use using the tool as a mediator for the effect). The 2SLS analysis shows that the effect is a bit higher than for the OLS analysis, namely, .16 of a standard deviation higher for students that practiced, compared with students that did not use the tool to practice. This corresponds to a small effect (see explanation above on Cohen's d). Note that in all four types of analysis in Table 3, there is hardly a difference between the model without and the model with covariates, also pointing at a successful randomization.

5.3 Heterogeneous effects

The results in Table 2, presented above, suggest that the effect of the ICT tool may be heterogeneous. Therefore, in these additional regression analyses, we allow the effect estimate to differ by distinguishing three ability groups. In this analysis, we incorporate interaction effects for whether a student scored low, medium, or high on the primary school ability test.

In these specifications, we include interaction effects between the treatment indicators and the performance group of the student, as well as the single terms for treatment and performance groups. This is done for each of the three performance indicators separately.

The results of these analyses are presented in Table 4. The first stages of these analyses were strong, with r‐squareds of .80, .79, and .92 and highly significant treatment dummies. We find a significant difference in the effect of the treatment indicator for students that scored low on the primary school achievement test and students that scored high on this test. This significant difference is in favour of the low‐scoring students, implying that students who displayed a lower ability level at the end of primary school, benefit more from practicing with the digital tool. We see a border significant difference (p = .10) between the middle and the lower groups, again in favour of the low‐scoring students. Although the upper group is significantly different from the lower performance group, the difference in coefficients results to zero, making us to conclude that there does not seem to be an effect for high‐performing students. This leads to the conclusion that the effect of the ICT tool is indeed heterogeneous and that low‐performing students benefit the most.