Journal of Computer Assisted Learning

2.1 Time on task

This work is motivated by the hypothesis that time on task provides a measure of a student's engagement during learning and that this measure is among the most important factors affecting learning and achievement (van Gog, 2013). The time‐on‐task hypothesis has deep roots in the science of learning, dating back to Ebbinghaus's (1885) classic studies showing a connection between time spent studying a word list and learning outcomes. In educational contexts, student engagement during learning—reflected in time on task—is often measured as the amount of time that students spend on learning (Carroll, 1963; van Gog, 2013). Through the years, the theory has evolved to incorporate the idea that engaged time on task, as opposed to simply time allocated to students for learning, is a more important factor in predicting positive learning outcomes (Karweit, 1984).

Recently, Rawson, Stahovich, and Mayer (2016) presented a basic model of academic learning that considered the fact that “engagement (as indicated by the amount of time that students allocate to a task) is a mechanism affecting learning outcomes (as indicated by achievement)” (p. 2). The researchers used digital smartpens to objectively measure time spent on writing homework assignments and found a significant positive correlation between this measure and final course grade. Our present research also considers the importance of time engaged in learning activities and seeks to collect objective measures of students' engagement with the assigned course e‐textbook.

2.2 Self‐testing as a learning strategy

Research has shown that academic learning depends not only on what is presented—such as a textbook—but on how the learner chooses to process the presented information, which can be called the learning strategy (Fiorella & Mayer, 2015). When students read a book—such as an e‐textbook in this study—they can control the reading process by choosing which pages to view and how long to view them. In regard to the types of pages summarized in Table 1, a student using a learning strategy based on learning by being told would focus heavily on the instructional text pages and the worked example pages, whereas a student using a learning strategy based on learning by practicing would focus heavily on the homework problem and homework answer pages.

Solving problems (i.e., learning by practicing) represents a form of self‐testing in which a learner engages in the kinds of activities that are required on the test, although learning by practicing is not always indicative of self‐testing. There is a considerable body of evidence that self‐testing can be an effective learning strategy (Dunlosky, Rawson, Marsh, Nathan, & Willingham, 2013; Fiorella & Mayer, 2015; Karpicke & Aue, 2015; Roediger & Karpicke, 2006), although only a fraction of the research base involves educationally relevant material (e.g., Johnson & Mayer, 2009). A common form of self‐testing involves studying a list of words and then trying to recall it. A testing effect occurs when students perform better on a subsequent recall test when they spend learning time trying to recall the words on a list rather than restudying them. The major learning mechanism underlying the testing effect is that learners strengthen their skill in retrieving the targeted material, which is a cognitive process that is also required on a subsequent recall test. In short, the act of taking a test—even a practice test—can be an aid to learning. This study extends research on self‐testing to the domain of learning from an online textbook in an academic setting, although solving practice problems may involve more than self‐testing.

More specifically, asking students to solve problems is consistent with the long‐standing research base on the positive role of practice with feedback in achieving expertise within a domain (Dunlosky et al., 2013; Ericsson, 2016; Hattie, 2009). In particular, certain aspects of practice have been shown to be particularly effective such as spacing practice activity over time rather than concentrating it all at one time (i.e., spaced practice effect), interleaving different types of problems rather than blocking problems by type (i.e., interleaved practice effect), and practicing on problems at an increasing level of challenge with appropriate feedback (i.e., deliberate practice effect). This study examines how students use online textbooks in the context of an undergraduate engineering course and explores on how students manage the way they practice solving problems.

On the basis of the time‐on‐task hypothesis and the self‐testing hypothesis, we focus on two key questions in this study: (a) Which kinds of pages in the course e‐textbook do students view during learning? If students see their goal as being exposed to the material—that is, learning by being told—they should direct their efforts towards the instructional text pages and the worked example pages. If students see their goal as being able to perform well on the homework problems—that is, learning by practicing—they should direct their efforts towards the homework problem pages and homework answer pages. (b) How are the learner's reading strategies (i.e., how much they engage in each of the four types of pages) related to academic achievement in the course? According to an updated version of the time‐on‐task hypothesis and the self‐testing hypothesis, the degree to which students focus on homework question and homework answer pages should correlate positively with academic performance, such as course grade.

3.1 Pedagogical considerations

Previous research has found that STEM students identify themselves as “readers” in surprisingly low rates. For example, just over 20% of students in an introductory physics course reported that they used the textbook for more than just referencing problems and equations (Cummings, French, & Cooney, 2002). Similarly, Podolefsky and Finkelstein (2006) conducted a survey of 1,000 participants across multiple introductory physics course offerings and found that only a little more than one third of the students reported reading more than 80% of the assigned readings. The same study also found that student's self‐reports of reading effort were not predictive of course performance or learning outcomes.

Previous research has speculated about possible factors affecting reported low usage of the textbook by students. For example, students have limited time to study many different resources and need to optimize study activities (Berry et al., 2010). If textbook reading is not directly tied to course grade, students may choose to study other resources they deem more beneficial to their grade, such as lecture materials (Podolefsky & Finkelstein, 2006). Other studies have found that the quality of the textbook and its content may affect students' willingness to use the resource. In one study, students' use of an e‐textbook in introductory physics courses varied greatly depending on whether or not the resource contained embedded assessments (Seaton et al., 2014). In another study, researchers surveyed students in introductory psychology courses and identified positive significant relationships between percentage of the textbook read and both the helpfulness of content (e.g., tables, figures, and examples) and quality of the writing (Landrum, Gurung, & Spann, 2012). When considering differences in course content, course structure, and student demographics, it is difficult to identify just how much time students should be spending with the textbook as many factors can play a part in students' decisions to study. Further, in many STEM disciplines, a heavy focus is made on applying concepts to solve problems. Often, the concepts are simple to explain and become clearer once applied to solve increasingly complex problems, and thus, it is possible that only a small amount of reading time is required to sufficiently learn the necessary material.

3.2 Textbook format considerations

The recent increase in use of electronic textbooks and open educational resources (OERs) has created a need to study the efficacy of these materials in relation to student use and learning. Recent research has indicated that even with a growing comfort with electronic media, students still prefer traditional textbooks to their electronic counterparts (Woody, Daniel, & Baker, 2010). Recent evidence suggests that readers interact differently with electronic and printed textbooks, but that media format does not affect student learning (Daniel & Willingham, 2012; Daniel & Woody, 2013; Rockinson‐Szapkiw, Courduff, Carter, & Bennett, 2013). Daniel and Woody conducted a controlled study of introductory psychology students' use of electronic and print textbooks for both in‐home and in‐lab conditions and found that e‐text users took twice as much time to read the content in both conditions. The authors investigated performance on a quiz based on the content and, for both conditions, found no significant differences based on media type. Similarly, OERs have grown in popularity as a more cost‐effective electronic alternative to traditional textbooks. Little research has been conducted to appropriately assess the effects this choice of resource may have on student study behaviours and performance (Griggs & Jackson, 2017; Gurung, 2017). In one carefully designed study, Gurung (2017) found that introductory psychology students spent significantly less time studying from OERs than printed text, a finding in contrast to studies examining electronic versus printed textbooks. Additionally, the author reported OER users scored lower on quiz questions drawn from the standardized advanced placement psychology exam.

The results from existing research clearly indicate that students use electronic resources differently than they use traditional printed text and that it is imperative to study this use to better understand the possible effects on student learning outcomes. The current research investigates students' use of an e‐textbook in an introductory mechanical engineering setting and aims to identify the relationships with students' course performance.

3.3 Methodological considerations

A potential methodological problem in much of the current research investigating students' textbook reading strategies is the reliance on surveys and self‐reported data (e.g., Berry et al., 2010; Cummings et al., 2002; Landrum et al., 2012; Podolefsky & Finkelstein, 2006; Schuman, Walsh, Olson, & Etheridge, 1985; Sikorski et al., 2002; Smith & Jacobs, 2003). This sort of data can be highly subjective and relies on retrospective reporting. For instance, Schuman et al. (1985) found little correlation between students' self‐reported study time and their grade. The authors speculated, “Students may not know how much they study, and there may also be some bias in willingness to report honestly” (p. 961). Additionally, Smith and Jacobs (2003) found a negative correlation between anticipated grade and self‐reported time spent using the textbook each week. In this case the authors concluded, “It is possible that a higher fraction of the weaker students consciously inflated their hours, or included inefficient time in their estimate” (p. 101). In our present research, we aim to overcome these methodological limitations by using computer‐based technology to accurately and objectively measure students' use of the textbook.

In recent years, the emergence of online content delivery systems (e.g., CourseSmart Analytics [https://www.vitalsource.com], OpenDSA [Shaffer, Karavirta, Korhonen, & Naps, 2011], LON‐CAPA [Kashy et al., 1993], and zyBooks [https://zybooks.zyante.com]) has provided one approach for measuring student reading effort. Studies using online content delivery systems to measure reading effort have found that students read at very low rates (Junco & Clem, 2015; Seaton et al., 2014). For example, one study examined the reading logs of thousands of students across multiple offerings of an introductory physics course (Seaton et al., 2014). In this study, reading time was assumed to be the time between subsequent page access events, but as a webpage can be displayed indefinitely, the researchers employed an upper bound on page viewing time. Assuming that meaningful page views had durations of between 10 s and 30 min, they found an average reading time of approximately 10 to 20 hr per student during the 15‐week term. Although weblogs provide a more objective measure of reading than surveys do, in this study, we developed an instrumented document viewing program that provides an even more precise method of measuring reading time.

4.1 Participants and course setting

The participants were 143 undergraduate students enrolled in a course on statics at the University of California, Riverside during the winter quarter of 2015. Every student who enrolled in the course consented to participate in this study. Statics is an introductory mechanical engineering course focused on the equilibrium of bodies subjected to forces. Mechanical engineering undergraduate students who follow the recommended course plan take this course during the winter quarter of the sophomore year. Table 2 presents demographic information for the participants. The vast majority of students were men (87%) and engineering or computer science majors (85%). Sixteen of the participants did not complete a survey soliciting their year in school, but of the remaining students, all but 9% reported being undergraduates within their first 4 years. We followed guidelines for ethical treatment of human subjects and obtained Institutional Review Board approval for the study.

4.2 Materials and apparatus

The materials comprised an online textbook and a document viewing program. The measures comprised reading measures and test measures.

4.2.2 Instrumented document viewer

To track students' reading habits, we created an instrumented document viewing program called “STL Reader.” The program was built using the open source μPDF (mupdf.com), PDF rendering software system. As illustrated in Figure 1, the viewer was designed for use on a Windows RT tablet and is typically operated in full‐screen mode. STL Reader provides functions for keyword search, bookmarking pages, and writing notes using either a stylus or a finger on the touch screen.

The user interface of the program is designed to enable accurate measurement of reading time. In particular, only a single page can be displayed at any given time so that there is no ambiguity about what page a student is viewing. Additionally, to ensure accurate measurement of engaged reading time, the program dims the display after 10 min of inactivity during which the student provides no input via the touch screen or keyboard. When the screen is dimmed, the student can “wake” the program by touching the screen. We provided students with the e‐textbook and other course documents as encrypted PDFs. The decryption password was built into STL Reader so that it could automatically display the documents. As students did not otherwise have access to the password, they could not view the course documents with other PDF viewers.

STL Reader creates a log file of time‐stamped page viewing events. An event occurs each time there is a change to the displayed document. For example, opening or closing a document, panning or zooming the display, and navigating to a new page are all recorded in the log. The program periodically uploads the log file to a secure file server.

Each student was provided with a Windows RT tablet on the second day of the quarter. The students were then required to register for an account and install STL Reader from the Windows Store. STL Reader then automatically retrieved an encrypted copy of the course e‐textbook from our secure file server.

STL Reader automatically uploads event logs to the server when the tablet is online. However, it is possible for a student to use the tablet without an internet connection. To ensure that all viewing data was collected, we manually extracted the log files from the tablets when students returned their equipment at the end of the quarter.

4.2.4 Test measures

We considered several test measures including homework and quiz score (based on 10 homework assignments and five quizzes), exam score (based on two midterm and one final exam), course grade (excluding class participation), Force Concept Inventory (FCI) score (Hestenes, Wells, & Swackhamer, 1992), and Statics Concept Inventory (SCI) score (Steif & Dantzler, 2005). We also administered a post‐study survey. The FCI measures students' understanding of Newtonian concepts of forces as taught in an introductory physics course, whereas the SCI measures understanding of the concepts taught in a statics course. All assessments (i.e., homework assignments, quizzes, and exams) contained statics problems requiring students to construct free‐form solutions using diagrams and equations and solve these equations to come to a final numerical solution; see Figure 2 for an example of a typical problem. The final exam also included two multiple choice questions and a professional ethics question.

Quiz and exam problems were graded using a rubric that examined the correctness of the major elements of the solution such as free body diagrams, geometric calculations, and equilibrium equations. The credit for the problem was divided over these elements according to their complexity, and points were deducted for errors. One problem on each homework assignment was also graded using this rubric scheme, whereas the remaining problems were graded on the basis of completion and correctness of the final answers. The problems assigned for homework were typically contained in the homework problem pages at the end of chapters in the textbook. Students solved these problems on paper with a digital pen and submitted the work electronically. The homework answer pages of the textbook contained final numerical answers for the problems, which enabled students to determine if they had solved the problems correctly. As noted above, however, the homework assignments typically specified changes to the numerical parameters of the problems so that the answers in the book did not match the problems as assigned.

The course grade was based on the following weighting: 5% for class participation, 10% for the homework score, 10% for the quiz score, 20% for the first midterm exam score, 20% for the second midterm exam score, and 35% for the final exam score. However, in our present analysis, we exclude class participation from the final course grade as this does not directly represent competence with the subject matter. For our short‐term measure of achievement, we combine homework and quiz grades with equal weights. Similarly, when considering the combined exam grade, we weight each midterm by 0.2 and the final exam by 0.35.

The post‐study survey included three questions related to reading activity. These questions, which have a 5‐point Likert scale, are as follows:

How important was the textbook to your learning of statics?

1. Very unimportant
2. Unimportant
3. No opinion
4. Important
5. Very important

How convenient was the tablet (STL Reader) for reading course documents?

1. Inconvenient
2. Somewhat inconvenient
3. No opinion
4. Somewhat convenient
5. Convenient

Do you prefer an electronic textbook or a paper textbook?

1. Strongly prefer paper
2. Prefer paper
3. No opinion
4. Prefer electronic
5. Strongly prefer electronic

4.3 Procedure

At the start of the class, students completed the FCI (Hestenes et al., 1992) to assess their prior knowledge of mechanics concepts. Each week, students attended 3 hr of lecture focusing on the core concepts of statics and 1 hr of discussion focusing on problem‐solving skills. Throughout the course, students were assigned weekly reading assignments and homework problem sets from the course e‐textbook. The reading assignments comprised instructional text and worked example pages from sections relevant to the week's lectures and homework. The homework assignments contained problems from the homework problem pages. In five of the weeks of the 10‐week quarter, the students completed quizzes with problems similar to the most recently submitted homework assignment. At the end of the course, students completed the SCI (Steif & Dantzler, 2005) and the post‐study survey.

5.1 Question 1: What do students read?

According to the self‐testing hypothesis, the most effective reading strategy is to practice solving problems by focusing on the homework pages. We quantified the reading behavior with three metrics—viewing time, number of page visits, and relative viewing time. The second column of Table 4 shows the mean viewing time (and standard deviation) for each of the four page types over the 10‐week quarter. Overall, students spent 26.3 hr (SD = 16.5 hr) on average reading the textbook (or 2.6 hr per week), including all types of content (i.e., instructional text, worked examples, homework problems, and homework answers). Due to violations of normality, we assessed the data using a non‐parametric Friedman test, with type of content as a within‐subjects factor and viewing time as the dependent variable, and found a significant difference among the four types of content, χ²(3) = 274.66, p < .001. Students spent the overwhelming majority of their reading time viewing homework problem pages (M = 22.1, SD = 13.4) and very little time reading instructional text (M = 1.9, SD = 3.1) and worked examples (M = 1.4, SD = 1.7).

The third column of Table 4 shows the mean number of page visits (and standard deviation) for each of the four page types over the 10‐week quarter. Due to violations of normality, we again assessed the data using a non‐parametric Friedman test, with type of content as a within‐subjects factor and number of page visits as the dependent variable, and found a significant difference among the four types of content, χ²(3) = 292.76, p < .001. Similar to the pattern for viewing time, there were far more visits to homework problem pages (M = 204.9, SD = 122.0) than for instructional text (M = 48.8, SD = 48.0) and worked examples (M = 27.2, SD = 25.7).

The fourth column of Table 4 shows the relative viewing time (and standard deviation) for each of the four page types over the 10‐week quarter. Due to violations of normality, we again assessed the data using a non‐parametric Friedman test, with type of content as a within‐subjects factor and relative viewing time as the dependent variable, and found a significant difference among the four types of content, χ²(3) = 274.66, p < .001. Students spent the majority of their time in the textbook viewing pages containing homework problems (M = 84.8, SD = 11.1) and a significantly smaller percentage of their time on instructional text (M = 6.8, SD = 7.6) and worked example (M = 5.1, SD = 4.7) pages.

We computed Pearson product moment correlations to investigate the relationships among the 14 reading measures listed in Table 3. Nearly all measures of viewing time and page visits were significantly and positively correlated with one another. Two exceptions to this pattern were found: viewing time and page visits for homework answer pages were positively, but not significantly, related to viewing time for instructional text. Table 5 presents the correlations among the four measures of relative viewing time. Relative viewing time for homework problems is significantly and negatively related to relative viewing times for instructional text and worked examples. Additionally, relative viewing times for instructional text and worked examples are significantly and positively related to one another. These findings illuminate a dichotomy in students' relative use of the textbook and further highlight the effect of students' choice of study strategies when viewing the textbook. Those students who tend to make greater relative use of the instructional text pages also tend to make greater use of worked example pages but not the homework problem pages. This pattern of behaviour is consistent with a student using a learning strategy based on learning by being told.

5.2 Question 2: How does student achievement correlate with what students read?

According to the time‐on‐task hypothesis, student reading effort (particularly on pages involving homework problems) should correlate positively with course grades. Table 6 shows the correlation between each of 14 viewing measures and course grade, exam grade, homework and quiz grade, and the SCI score, while controlling for the three measures of prior knowledge (high school grade point average [GPA], SAT score, and FCI score). Although all three types of measures (i.e., viewing time, page visits, and relative viewing time) show similar patterns, we focus on relative viewing time (in the bottom four rows) as an inclusive measure of selective viewing. Focusing on homework problems (measured by relative viewing time for homework problem pages) is significantly and positively correlated not only with homework and quiz grade but also with exam and course grades, whereas focusing on instructional text (measured by relative viewing time for instructional text) and worked examples (measured by relative viewing time for worked examples) is significantly and negatively correlated with all three grades. One should be careful to interpret these findings to mean worked examples and instructional text content are harmful for learning the material. In fact, there is a considerable amount of research speaking to the effectiveness of worked examples given the right conditions (e.g., Paas, Renkl, & Sweller, 2003; Renkl, 2002; Ward & Sweller, 1990). The pattern of results is consistent with the self‐testing hypothesis, which holds that practicing the activity required for the test is an effective study strategy. Similarly, this pattern extends the time‐on‐task hypothesis by highlighting the value of spending time on activities most closely related to the test activities.

Achievement on the SCI did not correlate significantly with any measure of reading effort (p ≥ .144 in all cases), perhaps because the e‐textbook did not specifically address the kinds of items on the SCI.

5.3 Supplemental questions

6.1 Empirical contributions

The first major finding of this study is that when college students in an engineering course view their e‐textbook, they tend to focus on homework problem pages (i.e., pages that allow them to practice solving problems), and they tend not to look at pages containing instructional text or worked examples (i.e., pages that tell them information). The second major finding of this study is that the amount of effort that students put into viewing homework problem pages correlates positively with achievement in the course, whereas the amount of effort that students put into viewing pages containing instructional text and worked examples correlates negatively with achievement in the course.

6.2 Theoretical contributions

The pattern of results is consistent with the self‐testing hypothesis, which holds that the activity of taking a practice test—such as solving homework problems—improves learning by providing a form of retrieval practice. The results also help to modify the time‐on‐task hypothesis to include the idea that academic achievement is related to effort on study tasks that are aligned with assessment (such as practicing solving problems when the assessment involves solving problems).

6.3 Practical contributions

Clearly, students in engineering courses do not put much effort into reading the instructional material in their e‐textbooks, skipping over a majority of the pages with instructional text or worked examples. This suggests that e‐textbooks in engineering should not be counted on to provide a rich base of conceptual knowledge for learners. Instead, students focus on the practical issue of completing homework problems, suggesting that e‐textbooks in engineering may best be used for building procedural and strategic knowledge in learners.

6.4 Methodological contributions

The STL Reader built for this project is an effective tool for assessing how students read e‐textbooks in academic courses. It has potential to contribute to future research on the relation between student study strategies and academic learning.

6.5 Limitations and future directions

This study involved one cohort in one course, so it would be useful to compare it with other studies. It is possible—although unlikely—that students used some resource other than the e‐textbook for instructional content, so future research should stringently monitor student use of external sources of information. However, the large amount of time students spent viewing homework problem pages provides strong evidence that students used the e‐textbook as their primary source for the textbook. At the time this study was conducted, the textbook was recently released, thus limiting the possibility of pirated versions being available online. Additionally, students were provided the electronic version of the textbook free of charge for use in the instrumented document viewing software. An additional way to minimize the possibility of students using a text other than the e‐textbook would be to employ a custom textbook designed specifically for the course. Because of the positive correlation between perceived convenience of the STL Reader and the amount of reading, care must be taken to ensure that document viewing software is carefully designed to match the needs of the students.

Additionally, the validity of the three self‐report measures should be interpreted in light of the fact that each is on a single item. We used only one item to tap each of the three self‐report factors because our main goal was not to develop a separate psychometrically tuned instrument for each factor but rather to take a preliminary look a few potentially interesting factors as a part of a supplementary analysis.

There is an abundance of research showing that a learner's interest in the subject matter can affect their engagement with learning activities (Renninger & Hidi, 2016). In an educational context, it is possible that those students who are more interested in the subject matter of the course will tend to read more of the instructional material and that this relationship may play a role in students' subsequent course performance. Although this is not the focus of this study, this is a rich area for further exploration.