Journal of Computer Assisted Learning

2.1 Blended learning and MOOCs

A subset of the online behaviour studies focuses on blended learning. Blended learning refers to “the range of possibilities presented by combining Internet and digital media with established classroom forms that require the physical co‐presence of teacher and students” (Friesen, 2012, p. 1). When the digital media consist of MOOCs, we speak of bMOOCs (Yousef, Chatti, Schroeder, & Wosnitza, 2015). Research on blended learning usually compares established classroom teaching with the MOOC part (e.g., Larson & Sung, 2009) and to a lesser extent examines variance within the bMOOC (Yousef et al., 2015). Comparative studies most often find no difference in learning outcomes between the MOOC and established teaching (Ashby, Sadera, & McNary, 2011). Sometimes, the online part leads to better results, especially in the field of STEM education (Vo, Zhu, & Diep, 2017). However, Conijn, Snijders, Kleingeld, and Matzat (2017) showed that even within a single institution, results may vary. Examples of measurements in this type of research are learning results, frequencies of logins and video views, forum use, resource use, and disposition variables such as perception and appreciation. Studies examining student perceptions report a preference for digital sources rather than traditional learning materials. This counts especially for high achievers (Owston, York, & Murtha, 2013).

2.2 Defining student performance in MOOCs

In MOOCs, various definitions are used to describe student performance. Often, researchers distinguish between completion and drop‐off, where completion is defined as students who submitted the final exam (Ramesh et al., 2014), students who showed behaviour in the last week (Sinha et al., 2014), or students who earned a normal or “with distinction” certificate (Allione & Stein, 2016; Jiang et al., 2014; Pursel, Zhang, Jablokow, Choi, & Velegol, 2016) and by self‐reports of (partially) completion (Adamopoulos, 2013).

However, MOOC completion may not be the best way to define student performance, because students in an MOOC have learning objectives and motivation beyond MOOC completion (Clow, 2013; Koller et al., 2013). Recently, a new typology has been developed to define student performance in MOOCs, based on initial intentions of the student (Henderikx et al., 2017). In this typology, students who do as much as or even more than they initially intended are considered successful. Accordingly, we define student performance as achieving the original intentions or learning objective(s) set by the student. In this way, we expect to find more useful pointers for the improvement of learning and teaching, rather than just boosting the MOOC completion rates.

Generally, it is hard to measure personal learning objectives because these have to be made explicit and might be diffuse. However, in the current study, learning objectives are available of students in the on‐campus course, a subsample of all MOOC participants. Therefore, we only include students who followed the MOOC as part of a blended on‐campus course, assuming these students have similar learning objectives, that is, completing the on‐campus course. Consequently, student performance is defined by the final exam grade for the blended on‐campus course.

2.3 Predicting student performance in MOOCs using general frequencies

Most studies predicting student performance in MOOCs focused on general frequencies of MOOC activities (e.g., de Barba, Kennedy, & Ainley, 2016; Ramesh et al., 2014). Overall, these studies found that being active in an MOOC has a positive effect on student performance. As videos are significant MOOC components, unsurprisingly, the number of videos watched is found to positively relate with completion of the course (de Barba et al., 2016; Pursel et al., 2016). Moreover, students who often replayed videos were 33% less likely to drop the course than students who rarely replayed videos. Students who watched significantly more videos than average were 37% less likely to drop the course compared with students who watched average or less than average proportions (Sinha et al., 2014).

Likewise, because quizzes and assignments are often required to complete an MOOC, activity in assessments is positively related with MOOC completion. Students who take the first quiz are 30% less likely to drop off, and students who take the peer assessment are 60% less likely, compared with students who only watch videos (Allione & Stein, 2016). The amount of quiz attempts is positively related with final grades (de Barba et al., 2016). Students who have a high average quiz grade are more likely to obtain a certificate (Jiang et al., 2014). Quizzes are mainly important predictors in the beginning and middle of the course (Ramesh et al., 2014).

Compared with videos and quizzes, posting on MOOC forums is a less robust predictor. Breslow et al. (2013) found that 52% of the students who gained a certificate were active on the forum. Others also showed that forum posts were positively related to MOOC completion, especially in the course's middle and end phases (Pursel et al., 2016; Ramesh et al., 2014). However, Allione and Stein (2016) found that students who posted in the forum were 28% more likely to drop off, and Adamopoulos (2013) found that having a forum is negatively related with completion.

2.4 Predicting student performance in MOOCs using more specific measures

The number of MOOC activities has shown limited value for predicting student performance and improving MOOCs. These aggregated activity measures are rather coarse and do not distinguish between different parts of the course. We expect that more specific measures of MOOC activity will result in both better predictions and guidance to improve MOOCs.

First, focusing on specific course items (e.g., watched video 1.1) could lead to better predictions, as the relation between course items and performance may differ across items. However, to the authors' knowledge, the effect of specific course items on student performance has not yet been studied. This may be because these measures are highly course specific, which results in low generalizability. Yet the effect of specific course items on student performance can provide a more precise indicator to improve specific course parts.

Second, several researchers found that sequences of interactions within videos are related to student performance (Brinton, Buccapatnam, Chiang, & Poor, 2015; Sinha et al., 2014). Using process mining to analyse an MOOC on process mining, it was found that students who passed showed a more linear path through the videos, but more random patterns of quiz submissions after submission of the first quiz, compared with students who failed (Van den Beemt, Buijs, & Van der Aalst, 2017). Thus, the order of activities in an MOOC also has an influence on student performance. Hence, in the current study, we extend on earlier research by including the effect of the order of specific course items on student performance, in addition to general counts of activities. On the basis of the findings by Van den Beemt et al. (2017), we also expect that students who passed will show a more linear path through the specific course materials (i.e., videos and literature) and a less linear path through quiz submissions, compared with students who failed.

Lastly, the number of activities and sequences of behaviour may depend on students' ability (measured by ACT and SAT tests) or past performance (Dollinger, Matyja, & Huber, 2008; Hattie, 2008), because these are robust predictors of student performance. Ability and past performance have been shown to be even better predictors than MOOC behaviour (Kennedy, Coffrin, de Barba, & Corrin, 2015). To determine the effect of MOOC behaviour on student performance in addition to past performance, we control for past performance, measured by prior GPA of the students in the bMOOC.

2.5 From predictions to actionable information and MOOC improvement

It has been shown that the analysis of MOOC use can guide teachers and course designers to improve their MOOCs. For instance, Guo, Kim, and Rubin (2014) analysed the time spent on videos to determine how these videos should be designed to improve engagement. Others analysed peaks of use within videos and suggested that automatic classification of these peaks could be used to label peaks, easing video browsing (Kim, Li, Cai, Gajos, & Miller, 2014). However, these studies did not relate the MOOC activities to student performance.

Taken together, we expect that predictions of student performance on specific course items and the order of activities in the MOOC give a better indication of what parts of the MOOC could be improved, compared with general measures (e.g., total number of videos viewed). However, these data do not allow to explain why certain behaviour occurred or how a specific part should be improved. It is important to first transform the data into actionable information, before it can be used to improve learning and teaching (Conde & Hernández‐García, 2015). To transform our data, results from the course evaluation are included that might explain some of our findings about MOOC behaviour. We expect this to result in actionable information.

3.1 Course design and participants

Data came from the graduate‐level on‐demand MOOC “Quantitative Formal Modeling and Worst‐Case Performance Analysis” provided on Coursera (https://www.coursera.org/learn/quantitative‐formal‐modeling‐1). The MOOC was part of the blended graduate course “Quantitative Evaluation of Embedded Systems” taught from November 9, 2015, to January 31, 2016. MOOC data were collected from a week before the on‐campus course started to the end of the exam period.

The on‐campus course consisted of the MOOC and face‐to‐face lectures. The on‐campus course focused on formalisms (especially dataflow graphs and Markov chains) used when quantitative aspects such as time, probability, and resource usage play a role in system behaviour analysis. The course was provided by three technical universities in the Netherlands (situated in Delft, Twente, and Eindhoven). The course was compulsory for students following the graduate program Embedded Systems and optional for other interested students. In total, 199 students registered for the on‐campus course, 156 students completed the final exam, and 125 passed the course.

The on‐campus course consisted of eight lecture weeks and two exam weeks (see Figure 1). The first two lecture weeks were used for the MOOC. The content of the MOOC was assessed with a practical assignment in Week 5 and the final exam in Week 9. Students did not have to complete the MOOC to pass the on‐campus course.

The MOOC consisted of 47 videos, 14 resources, 9 graded quizzes, 5 practice quizzes, 1 peer‐reviewed assignment, and a discussion forum, divided over four modules. The MOOC was relatively small with 2,123 students enrolled, whereas a typical Coursera MOOC consists of 40,000 to 60,000 students (Koller et al., 2013). Only 30% of the students started a video (compared with the typical 50–60%), 16% started a quiz, 5% started a peer‐reviewed assignment, and 2% posted on the discussion forum. The completion rate of 2% is rather low compared with the current average completion rate on Coursera of 6% (Jordan, 2015). These low numbers can (partly) be explained by specificity of the MOOC, the relative high level (graduate level), and the short time frame in which data were collected.

In the current study, only students (n = 199) enrolled in the on‐campus course are considered. The assumption is that these students have similar learning objectives (completing the on‐campus course).

3.2 Data preprocessing

Several data sources were used to analyse the on‐campus course: MOOC data from Coursera, performance data, and course evaluation data. Data preprocessing and data integration were done using the statistical package R.

3.3 Data analysis

After data preprocessing and data integration, the MOOC activity frequencies were analysed using Stata 14. Descriptive analyses showed that there was little activity in the peer‐reviewed assignments and the forum. Therefore, the variables peer assignment finished, peer assignment reviewed, forum questions posted, and forum answers posted were transformed to binary variables (0: zero activity, 1: any activity).

Subsequently, Pearson correlational analyses of all variables with final exam grade were conducted. Thereafter, a multiple linear regression using backward stepwise regression was run, using the aggregated measures of MOOC activities (with and without past performance). All predictors with a p value >.2 were removed from the model. Robust regressions were used, because the assumption of homoscedasticity was often not met. The Stata function “crossfold” was used for tenfold cross‐validation, which runs 10 regressions on subsamples and takes the average of these regressions (Daniels, 2012).

The order of activities was analysed using ProM 6.6 (ProM, 2016), a process mining tool. First, a dotted chart was generated to determine the activity in MOOC modules during the on‐campus course, for the different final exam grades (similar to Van den Beemt et al., 2017). Second, two models were made with Fuzzy Miner, a ProM plug‐in, to compare the sequences of activities between students who passed and students who failed the on‐campus course. The Fuzzy Miner simplifies the view of a process model by using abstractions based on the significance of activities and relations between these activities. Significance refers to the relative importance of the activities and can be measured in multiple ways (Günther & Van der Aalst, 2007). Here, unary significance was used: the more an activity occurred in the log, the higher the significance. Because our main goal was to determine the differences and not to obtain the best model fit, the default settings of the plug‐in were used.

3.4 Preliminary analysis: Differences between whole sample and subsample

Past performance was only available for a subsample of the students. Because this subsample was not randomly chosen, t tests and multiple linear regressions were used to verify similarity with the whole sample.

Independent samples t tests showed that the outcome variable exam grade and most MOOC activity frequencies did not significantly differ between the students with and without past performance data available. Only the use of the peer‐reviewed assignment showed significant differences: Fewer students finished the peer‐reviewed assignment, t(89) = 2.01, p = .05, and fewer reviewed the peer‐reviewed assignment, t(89) = 2.15, p = .03, in the subsample compared with students outside the subsample. Hence, we cannot generalize the use of peer‐reviewed assignments of the subsample to the whole sample.

Simple linear regression showed that being in the subsample did not have a significant influence on exam grade. Multiple linear regression showed that being in the subsample still did not have a significant influence on final exam grade when all predictors (MOOC activities) were added. Lastly, when interactions of all predictors with being in the subsample were included in the regression, none of the interactions were found significant. Hence, there is no difference between the effects of the predictors on final exam grade between students with and without past performance data available. This indicates that the effects of the predictors on final exam grade from the subsample with past performance available can be generalized to the whole sample.

4.1 MOOC activity frequencies

First, MOOC activity frequencies were analysed. From the 91 students, most started and finished at least half of the videos, started at least half of the quizzes, and read at least half of the resources. On average (median), the students finished 46 of the 47 videos, passed seven of the nine graded quizzes, and read five of the six resources. The quiz grades were relatively high: higher than 8 (out of 10) on average. An explanation could be that students needed an 8 to pass the quiz and could attempt a quiz multiple times to improve their grades. Almost half of the students finished and reviewed the optional peer‐reviewed assignment (43 and 44, respectively). The forum was rarely used: 14 students posted questions, and 18 students posted answers.

Correlational analyses were conducted between the MOOC activity frequencies and final exam grade scored in the on‐campus course (see Table 1). The MOOC did not cover all topics of the final exam. Nonetheless, all activities correlated significantly with final exam grade, with small to moderate effect sizes. The number of videos watched, and quizzes passed and finished correlated positively with final exam grade (rs = .39–.54, p < .001), whereas the number of graded quizzes failed correlated negatively (r = −.28, p < .01). Both starting and finishing the peer‐reviewed assignment had a positive correlation with final exam grade (r = .36, p < .001; r = .22, p < .05). Posting questions or answers on the forum was positively correlated with final exam grade; however, the effect sizes were small (r = .24, p < .05; r = .30, p < .01).

When looking at the correlations for the students with high and low past performances separately, almost none of the activities correlated significantly with final exam grade. This is probably due to the low sample size. Interestingly, there was a negative trend for the number of videos watched on final exam grade for students with low past performance, but a positive trend for students with high past performance. The peer‐reviewed assignment showed a significant positive correlation with final exam grade, but only for students with low past performance (r = .69, p = .01). Thus, being active in the MOOC might not lead to higher final exam grades for all students.

To predict final exam grade, multiple linear regressions were run with final exam grade as outcome variable and all 21 MOOC activity variables as predictors. The final model after backward stepwise regression consisted of quizzes finished, graded quizzes passed, unique resources read, peer assignments started, forum answer posted, unique videos finished, and peer assignments finished (Table 2, Model 1). Combined, these variables could explain 52% of the variance in final exam grade (cross‐validated R² = .46). Thus, although the final exam did not only cover the content of the MOOC, MOOC activity could already explain a high amount of variance in final exam grade. Moreover, all different MOOC activities (videos, quizzes, peer‐reviewed assignment, resources, and the forum) explain a unique part of the variance.

We expected that past performance would have an influence on the effects of MOOC activities on final exam grade. Therefore, in the second model, past performance was added as control variable. The final model consisted of videos started, quizzes finished, (unique) resources read, peer assignment started and finished, and past performance (Table 2, Model 2). Combined, these variables accounted for 75% of the variance in final grade. Thus, adding past performance improved the prediction accuracy. However, it should be noted that past performance data were only available for 26 students, resulting in a small sample. With this many variables and small number of students, the model could easily be overfitted. Unfortunately, this could not be determined, because the sample size was too small for tenfold cross‐validation.

4.2 Specific course item frequencies

Next to the general activity frequencies, specific course item frequencies were analysed. Highlights of the use per specific course item can be found in Table 3; the complete table is presented as Table A1. On average, videos were started 110 times by 80 unique students and finished 233 times by 76 unique students. This means that the beginning of videos (first 5 s) was watched less often than the end of videos (last 5 s), whereas only a few students dropped off during videos. Thus, students more often replayed the end of videos, perhaps because this part was more interesting or difficult than the beginning. As expected, the number of students who watched a video dropped over time, indicating that some students dropped off during the course.

On average, each video was watched 2.18 times. Especially the videos at the end of Module 3 were replayed more frequently. This may indicate that these videos are more difficult to understand, more interesting to watch, or are needed for a specific assignment. The quizzes, and especially the graded quizzes, were attempted more often than the videos. One quiz “Calculating optimal periodic schedules and their latencies” was even attempted more than 13 times, on average. All resources were accessed mostly only once.

Correlational analyses between the specific course items and final exam grade showed that 65% of the course items correlated positively with final exam grade. All quizzes, except the first quiz, were significantly related with final exam grade, with a small to moderate effect size (rs = .20–.40, p < .05). Moreover, the effect sizes increased during the course, up to effect sizes of .54 at the end of Module 3. This indicates that course items later in the course have more effect on final exam grade, perhaps because some students already dropped off by then or these materials were assessed more in the final exam. The resource, “materials created by fellow students,” a course summary made by one of the students, had the highest effect size (r = .59, p < .001).

Specific course items frequencies were not used to predict student performance, as these frequencies show high collinearity and the number of variables would be too much for reliable prediction for this relatively small sample size.

4.3 Order of MOOC activities

Lastly, the order of MOOC activities was examined. First, a dotted chart was generated to determine whether the students used the content in the provided order and timing of the MOOC (Figure 2). The results showed that the students in general stuck to the provided order. In the first week, they primarily used course items from the first module in the MOOC. In the second week, students mainly used items from Module 2 and occasionally looked back to resources provided in the first module. Thereafter, students accessed course items of the third module but spread out over 2 to 3 weeks. Occasionally, course items from the first two modules were (re)visited. Thereafter (just before the Christmas break), course items of the last module of the MOOC were increasingly used. After the Christmas break, course items of all modules were revisited, indicating that students were reviewing all content in the last week(s) before the exam.

The dotted chart shows some differences in the order and timing of the activities between students who passed and students who failed the final exam. On average, students who passed the exam started earlier with the content of the second module, compared with students who failed. Moreover, the students who passed the exam showed more constant activity in the MOOC during the whole course, compared with students who failed, who almost did not use the MOOC in the third and fourth weeks of the course. Lastly, students who passed the final exam more often revisited the course content in the last week(s) before the exam, revisited more course items, and started earlier with revisiting the course items, compared with students who failed. Thus, passing students spread their learning over more days compared with failing students.

To determine differences in the order of specific activities between students who passed and students who failed, two fuzzy models were created for both groups. For simplicity, only finished activities were included, using the first time the activity was finished (see Figure 3). For readability, the activities that show the same linear order as in which they were provided in the MOOC are merged and represented by a dotted arrow.

Both models show that most students started with the introduction video and followed the provided order of activities at least until half of the videos in the first module. Thereafter, some students started to skip the quizzes (e.g., “Modeling warehouse 13”). The activities in Module 2 were watched in linear order, whereas activities in Modules 3 and 4 were accessed less linearly: Students started skipping videos and quizzes and accessed some of the videos and quizzes in a different order than provided in the MOOC.

When comparing the models for students who passed with the students who failed, a few differences can be found. Students who passed the final exam showed somewhat higher significance for most activities; thus, less activities were skipped, compared with students who failed. Contrary to the students who failed, some students who passed did not start with the introduction video; they seemed to go straight to the content they needed. In general, though, there is little difference between the order of the activities for students who passed and failed. Thus, there seems to be little relation between the order of activities in this MOOC and student performance in the on‐campus course.

5.1 Limitations and future work

This study knows several limitations that in turn can be used as starting point for future research. First, there were limitations in the data obtained from the MOOC. Data were available about the start and completion of videos, indicating whether the student watched the first and last 5 s, respectively. Here, we assumed that when students started and completed a video, they watched the whole video. However, it is unknown if the students watched the whole video or skipped the middle part. Future work should include clickstream data (Brinton et al., 2015; Sinha et al., 2014), to have a more reliable measure of video use. Additionally, clickstream data could further specify improvements for specific parts within videos, for example, by automatically labelling highly used parts in the video, to ease video browsing (Kim et al., 2014). Due to privacy policies, we were not able to use and combine all available data sources. Therefore, the sample sizes are limited. However, preliminary analyses showed that the sample with past performance available can be generalized to the whole sample.

Second, we assumed that all on‐campus students had similar learning objectives, which turned out to be an oversimplification. Yet the learning objectives of on‐campus students are more similar than the learning objectives of all students in the MOOC. In future work, learning objectives could be measured in the form of a prequestionnaire (Henderikx et al., 2017) and perhaps also during the MOOC, considering that learning objectives might not be known from the beginning and might change over time. With learning objectives specified, it could be determined how the behaviour in MOOCs differs across students with different learning objectives, which in turn can be used for more personalized interventions.

Third, we only included past performance as background information. However, other background information has been shown useful to improve learning and teaching as well. For example, learning dispositions are highly valuable for timely feedback generation (Tempelaar, Rienties, & Giesbers, 2015). In addition, self‐regulated learning could be measured to support students' control of their own learning processes in MOOCs (Jansen, van Leeuwen, Janssen, Kester, & Kalz, 2017). Future work should also include these background variables, to obtain a more complete understanding of students' behaviour in the MOOC, which could be used for more personalized improvements and student support.

5.2 Practical implications

In this study, we showed that MOOC activities can be a reliable indicator of student performance in the blended course. This is in line with literature on blended learning in MOOCs, which shows that there is often no difference in learning outcomes between the MOOC and established learning (Ashby et al., 2011). Thus, to improve learning outcomes, it is equally important to improve the MOOC as improving traditional education and learning materials.

To improve an MOOC based on the prediction of student performance, several points need to be considered. First, the definition of student performance has an influence on the implications for MOOC (re)design (cf. Henderikx et al., 2017). For this, the goal of your MOOC (re)design needs to be considered: Do you want to improve MOOC completion rates, or do you want to aid students in achieving their learning objectives?

Second, the indicators used to assess student performance have a large influence on identifying the points of improvement in the MOOC. We showed that specific course item frequencies or the order of activities can provide more insight into where the MOOC needs to be improved. The specific course item frequencies can show where problems can be expected. For example, the high number of replays of the end of the videos indicates that the content on the end of the videos needs to be reconsidered and if possible explanations or more practical examples should be added. In addition, the positive relation between performance and students making and sharing their own materials online can be read as a call to stimulate students to actively summarize learning content and share it with their peers. This would offer another way to interact with the content and learning materials.

Third, the addition of other proximal data, such as motivation, past performance, and future plans, could lead to more personalized and adaptive improvements in the MOOC. Lastly, it is useful to add additional information such as course evaluations. Course evaluations do not necessarily relate to student performance; thus, it is necessary to be cautious to use the evaluations directly to improve the MOOC (Rienties & Toetenel, 2016). Yet they can be useful to interpret the analyses on MOOC behaviour and translate them into actionable information.