The goals of evaluation of computer-assisted learning (CAL)11 materials include providing evidence whether such materials enable students to learn effectively and improving the design of programs to facilitate student learning . CAL materials have been used for teaching biochemistry for many years, and over this period of time there has been considerable debate as to whether students learn more effectively, or more deeply, from multimedia or CAL programs. A review of 355 studies showed no significant difference in the effect of CAL in enhancing learning . Others have reported enhancement of learning under specific conditions . In our teaching we have observed that some students performed well in their summative biochemistry examinations but performed poorly in specific content areas that were supported by CAL materials and vice versa. The aim of this qualitative study was to explore students' approaches to learning with a CAL program and the resultant learning outcomes.
This article reports on a qualitative evaluation of student use of a computer-assisted learning (CAL) program and associated learning outcomes. Learning outcomes were classified by analysis of examination scripts using the Structure of Observed Learning Outcomes (SOLO) taxonomy, and individuals were interviewed to uncover aspects of their learning processes while using the CAL program. All students were aware of the “design for learning” features of the CAL program, although not all students were able to take advantage of these because of their own note-taking practices. Verbatim or excessive note-taking was associated with surface learning outcomes, while summarizing or personal note-taking and engagement with the program was associated with deep learning outcomes. Other factors affecting learning outcomes included students' perceptions of the role of the program, social elements of the learning environment, and a lack of distinction between major and minor issues in the content.
THE TEACHING CONTEXT
The Biochemical Basis of Anemia was a CAL program used in a university context for teaching a large class (n = 220) of undergraduate health science students. Teaching strategies for the study unit on the biochemistry of anemia consisted of two lectures, one small group tutorial, and unlimited use of the CAL program. In essence the content of the CAL program was equivalent to that of the lectures and tutorials combined, and students could choose to attend any or all activities. The Biochemical Basis of Anemia program consisted of five modules: (i) description of the anticipated student learning outcomes, (ii) biochemistry of iron, (iii) folate and (iv) cobalamin (vitamin B12) each with embedded questions, and (v) case studies. The Iron module covered types of iron in foods, factors affecting absorption of iron, daily iron balance in men and women, and mechanisms and time frames for development of iron deficiency. The Folate module outlined the various forms of dietary folate and mechanisms of absorption and illustrated their inter-relationships and the role of folate in major biochemical reactions using colored carbon groups in structural formulae. Similarly, the Cobalamin module used colored carbon groups in structural formulae to illustrate its role in biochemical reactions. The two case studies focused on the anticipated consequences of iron deficiency arising from gastrointestinal bleeding and cobalamin/folate deficiency resulting from a deficiency of intrinsic factor.
The biochemistry modules provided a general survey of the content area, deconstruction of the complex nature of the content into manageable chunks, the sequential development of biochemical structures and pathways leading to construction of the complex “big picture,” interactive questions, and opportunities for linkage to prior learning and reflection. The navigation of the modules was semi-linear; thus previously encountered diagrams and structures of importance were available from pull-down menus for later use in attempting the case studies. The students were encouraged to apply their biochemical knowledge about iron, folate, and cobalamin in the context of two clinical cases and to develop links between patient symptoms, biochemistry, and pathology.
INDICATORS OF EFFECTIVE LEARNING
Ideally indicators of effective learning need to reveal cognitive processes and a measure of deep learning outcomes. An instrument developed to reveal students' habitual approaches to learning tasks is the Study Process Questionnaire (SPQ) . This questionnaire uses self-report questions to identify student tendencies toward taking a deep approach (looking for meaning), a surface approach (rote learning facts and procedures), or an achieving approach of doing whatever the student perceives will get the best grades. Patterns of scores by undergraduate science students on these aspects provide an indicator of cognitive processing while learning .
The outcomes of learning can be used to reveal cognitive processes by conducting an analysis using the Structure of Observed Learning Outcomes (SOLO) taxonomy . This taxonomy (Table I) is applied to an open-ended question answered after learning on the topic is complete. The taxonomy is used to identify whether the student has determined a meaningfully related structure in the newly learned material and whether this has been extended by relationship with prior knowledge related to the topic or whether the learner has reproduced a number of points without relating them in a meaningful way.
METHOD OF EVALUATION
The aim of the research was to explore the factors that influenced depth of learning with use of the CAL program. To classify the depth of learning outcomes, the SOLO taxonomy was applied to the students' examination question on the topic covered by the CAL program. Coded examination scripts completed by participants (n = 36) were scrutinized independently by both authors, and their answers to the CAL question were rated according to the SOLO taxonomy (Table I) to identify the extent of deep learning. Answers categorized as relational or extended abstract were deemed to indicate deep learning. The SPQ was used to identify the potential of the individuals within the group for deep learning outcomes. Students who agreed to participate in the study completed the SPQ questionnaire in a tutorial session. A comparison of scores on the two instruments indicated whether the students were consistent with their habitual approach to study and in their approach to this learning task involving the use of CAL. To identify factors that may influence the students' approaches to learning with CAL, those students who were not consistent between their Deep Approach score on the SPQ and the depth of learning as an outcome identified by SOLO (Table II, Groups 2 and 3) were selected for a semistructured interview .
Scores on the SPQ revealed that 20 students habitually used deep learning strategies, and 16 students habitually used surface learning strategies in their studies. Deep learning of the CAL topic was demonstrated by the majority (n = 23) of students who answered the examination question in a manner that related the biochemistry of the various vitamins to pathology, clinical signs, and symptoms and were rated as achieving a relational (deep) learning outcome (Table I). Approximately one-third of students (n = 13) were rated as achieving a surface learning outcome because they failed to outline relationships between the metabolism of the vitamins and/or link these to pathology or clinical signs and symptoms of disease. There were 15 students that showed inconsistencies between their usual approach to study and their learning outcomes (Table II). These students with habitual deep learning approaches but surface learning outcomes from the CAL (Group 2, n = 6) and with habitual surface learning approaches but deep learning outcomes from the CAL (Group 3, n = 9) were selected for interview. Highlights and representative comments from the 15 interviews are outlined below.
General Comments on CAL Program from Student Interviews
The specific influences of the CAL program on the students' learning were revealed in the interviews. Most of the students found that the CAL program brought the topic together and helped them to see how all aspects of the topic were related. They found that the CAL structured and presented the material better than the lectures and that it aided their learning by giving them an overview of the topic. Comments such as “the CAL helped to clear up things I didn't understand in the lecture,” “it sort of put it all together,” and “the CAL is more in depth and clarified” are typical expressions of this point. The students tended to find the program challenging and thought-provoking as illustrated by the comment “you really had to think” in relation to the questions. Others saw it in terms of an active learning process that enabled them to reach a more integrated understanding of the topic. One comment that it “helps you to understand the basic principles” demonstrates an insight into depth of thinking on the part of one student.
Deep Learners That Demonstrated Surface Learning of the CAL Topic (Group 2)
Abraham demonstrated a high level of awareness of his own mode of study and was able to place the value of the CAL program clearly in this context. Abraham used the lecture notes, tutorial, and CAL program to study the topic. He found the CAL program to be a valuable source of information that was well organized. Abraham made the point that he tried to understand material first but admitted that his note-taking from the CAL programs was almost verbatim. In relation to forming a deep understanding of the topic, Abraham said “with the CAL program I have time to think about what's going on from one screen to another to understand it.” However, when pushed further on the issue of deep understanding he equated deeper understanding with more information. Indeed Abraham showed some insight into his own “passive” learning processes and said the CAL program design pushed him to become actively engaged.
Denise took her study seriously and followed every available avenue to develop and clarify her understanding of the topic: lecture notes, the textbook, tutorial, CAL program, and discussions with the lecturer. She saw the CAL program primarily as a source of information and described intensive use of the program: “I practically took dictation from the screen. I just took down almost everything.” In her interview Denise indicated an attempt to understand but with a major emphasis on note-taking from the CAL program. She also took notes from the textbook and wondered if the CAL program would have been just as valuable in print. An interesting aspect of Denise's interview was her own realization that she was not using the program as it was intended. When advising students how to use CAL, faculty say “don't just take down everything that was on screen, sit there and enjoy it, take it all in, but I don't remember the information doing it that way.” The interview highlighted both the strength of her motivation and a major emphasis on memory. She ensured success by using every strategy to compile information and understanding rather than exploiting the interactive operations of the CAL program to help her learn.
Surface Learners Who Demonstrated Deep Learning of the CAL Topic (Group 3)
Joanne appeared to have mixed feelings about the CAL program as an aid to learning. She attached primary importance to the textbook and tutorial, but upon subsequent reflection the CAL program emerged as being critical in the development of an understanding of the topic. Joanne made her own notes when using the program but only noted the main points in each frame. Joanne emphasized the importance of the textbook in giving her a knowledge base with which to use the CAL program. However, when asked about thinking deeply on the topic Joanne started to see a greater value in the CAL program as opposed to the text: “the CAL program helped most because it showed quite a few things, then you really had to think to make the relationship between the things it was showing you, whereas in the textbook, you just sort of read about it and its like a lot of facts … but it didn't help with tying things together.” She believed the clinically oriented problems did this (integration) because they required solutions where you “have to put more thought behind the learning that you do.”
Caroline saw the CAL program as having a specific role of being complementary to the lecture and an important part of the learning process on the topic. Caroline worked from lecture notes and the CAL program; she didn't use a textbook. She saw the CAL as part of an overall teaching strategy in that “it supplemented what we need rather than giving us a whole lot of new information.” Caroline observed that taking extensive notes actually interfered with the value of the CAL program as a learning aid. She felt that the information was not the main value of the CAL program, but verbatim note-taking could clearly distract the learner.
There were two major findings from this study. First, students found the CAL program to be a useful learning aid, and areas for improvement were identified through interviews. Second, the nature of note-taking and learning approach to the CAL program appeared to correlate with learning outcomes.
Views on the CAL Program—
All students interviewed indicated a general preference for the CAL program over the lectures, tutorial, and available textbook. There were two points of similarity heard in most interviews. Students showed a clear appreciation of the interactive nature of the program and the way the CAL program helped them to see relationships between concepts and metabolic pathways. They also commented on the importance of the embedded questions for promoting deeper learning because the answers were not easy and immediately obvious. Most students also showed an awareness of how the program was designed to enhance learning and a self-awareness of ways in which they learn. Individual students identified “design for learning” features such as the sequential and logical development of concepts, the stepwise construction of diagrams and structures, and the use of questions to promote thinking and engagement with the program content. Some students found that the linear and semilinear nature of the program design presented a barrier to their understanding because it did not clearly distinguish between the hierarchy of information: primary concepts or main point, minor points, and examples. Similar observations have been made from students learning from reading of text materials  and viewing television  where individuals might recognize important points but not understand or “see” their relative importance. Although structuring and organizing approaches were used (e.g. headings, subheadings, and sign-posting), more features to emphasis the hierarchal structure of information, such as a concept map or pop-up windows with diagrams or flow charts to emphasize key points, will be considered in a revised program.
Learning Approach and Learning Outcomes—
The SPQ was used to determine each student's usual learning approach, while the SOLO taxonomy was used to distinguish between deep and surface learning outcomes on the CAL topic. In this study there was a correlation between learning approaches and learning outcomes in 60% of the student group (Table II), i.e. deep learning approach with deep learning outcome. These findings are consistent with those of Boulton-Lewis  who reported a positive relationship between student scores on the SPQ and use of SOLO on assessment items. Boulton-Lewis found that students with a more structured organization of knowledge were less concerned with surface motives and strategies and more concerned with deep strategies than those with less organized conceptions of learning. But what of the other 40% of students where there was no correlation? It is worthy to note that while the SOLO taxonomy reveals the nature of the student's thinking and conception of the topic, it does not reveal the learning strategies that were used. Similarly, the SPQ, while revealing the student's usual approach to study, also does not provide any indication of the approach used in specific instances. The interview was used to explore these issues and revealed that an important difference between the deep and surface learners was their approach to note-taking from the CAL program.
Note-taking and Learning Outcomes from CAL—
Note-taking from the CAL program was a learning strategy common to both deep and surface learners. However, excessive note-taking from the CAL program was associated with surface learning outcomes, whereas brief note-taking, or summarizing, was a feature of the approach used by students that demonstrated deep learning outcomes. Comments such as “I practically took dictation from the screen” and “I have to write everything down” were representative of the surface learning outcome group (Group 3), many of whom indicated that this was necessary for them to “remember” what was in the program. In addition they tended to see the CAL program primarily as a source of information. The deep learning group also took notes, however, they took the form of a summary or were limited to new information. Brian “read it first … and whatever I got out of it, I just write down”; Alistair shared note-taking with a friend, and they “took down the main points,” while Caroline “took notes on things that I didn't know.” Each of these students saw the use of the CAL program as one of many sources of information and as a means of reaching understanding.
Note-taking is perceived as an important part of learning, particularly with lectures and tutorials, and much of the research in this area may be applicable to learning with CAL. Students may learn through the process of note-taking alone, although the availability of notes for study and the generation of relationships between concepts are more important that the note-taking process alone . Extensive note-taking from the CAL may have distracted students from any learning opportunities afforded by the CAL program, and some students were aware that they “probably didn't think much while I was doing” the CAL program despite embedded strategies to foster interactivity. The majority of students in this study did not see note-taking per se as important, but they all recognized the importance of their notes for study and review. Embedding mandatory note-taking in CAL programs has led to improved knowledge recall (compared with non-note-takers) . However, modeling note-taking through the use of cognitive tools  such as notepads and helping students to see relationships through concept mapping tools embedded in CAL programs may encourage deeper engagement and lead to enhanced learning outcomes.
Various other reasons why these students adopted an approach different from their usual learning approach when using the CAL program emerged from the interviews. Abraham, normally a deep learner, also regarded the CAL program as a source of information but said that his inability to differentiate between important concepts and examples or minor issues appeared to promote excessive note-taking. Denise and Janette, normally deep learners, regarded the CAL program primarily as a set of notes to copy down for reference because it helped draw the concepts together into a coherent whole. By contrast, Brian and Hung, normally surface learners, valued the CAL program as a tool to help them deeply understand the topic. While for Alistair, typically a surface learner, the opportunity to discuss information with colleagues may have promoted deeper learning and understanding so that fewer notes were needed than usual. Clearly, factors relating to the educational design of the program, the learning environment, and student perception of the role of the CAL all impacted upon learning outcomes. Since there are many different designs and learning environments used in teaching, this may explain why the evaluation of CAL has yielded unequivocal results . This study provides further evidence that one challenge in CAL for academics and educational developers is to use knowledge of how students learn and knowledge of the technology to design learning experiences that promote a deep approach to learning. Another major challenge is to induct students into better levels of engagement with CAL through improved CAL design and by providing teaching environments that maximize learning opportunities from CAL.
This CAL program can be seen to have been effective in aiding students' learning and in providing some encouragement for the students to take a deep approach to their learning. The value of the program in integrating the students' learning on the topic also seems clear. Extensive note-taking from the CAL program was associated with surface learning outcomes, whereas brief note-taking was associated with deep learning outcomes. Other factors affecting learning outcomes included students' perceptions of the role of the program, social elements of the learning environment and organizational structure, and labeling of content. CAL program designers and teachers using CAL programs should consider embedding tools that increase active engagement with information presented in the CAL program and offer intended ways of using such programs.
|SOLO level||Description||No. of students|
|Question: Describe the clinical consequences of a deficiency of the vitamins folic acid and vitamin B12 referring to the biochemistry underlying the clinical signs of deficiency, the metabolic inter-relationships between the two vitamins, and characteristics distinguishing deficiency of these vitamins from other common nutritional deficiencies.|
|Prestructural||No competence in the task is demonstrated (e.g. B12 and folate deficiency both cause anemia).||2|
|Unistructural||Only one relevant aspect of the task is referred to (e.g. folate deficiency results in decreased synthesis of TMP, which in turn affects the synthesis and maturation of red cells causing anemia).||1|
|Multistructural||Several relevant aspects are referred to without any clear relationships between them being identified (e.g. folate deficiency results in decreased synthesis of TMP, which in turn affects the synthesis and maturation of red cells causing megaloblastic anemia. B12 deficiency results in decreased synthesis of TMP, which in turn affects the synthesis and maturation of red cells causing megaloblastic anemia).||10|
|Relational||The student has integrated elements of the response into a meaningful structure (e.g. folate and B12 are cofactors for different enzymes on a common pathway leading to the synthesis of TMP (pathway outlined). Deficiency of either or both of these vitamins may decrease the availability of dTMP required for maturation of red blood cells. This deficiency results in anemia characterized by the presence of megaloblasts in the marrow and larger sized red cells).||23|
|Extended abstract||The student has elaborated on the response by relating it to other topics and his/her prior knowledge and experience (e.g. as above plus reference to fact that iron deficiency also leads to microcytic anemia with no megaloblasts).||0|
|Group||Category||No. of students||Interview|
|1||Students rated as deep learners on the SPQ who responded at a deep level to the test question as rated by the SOLO taxonomy.||15||No|
|2||Students rated as deep learners on the SPQ who responded at a surface level to the test question as rated by the SOLO taxonomy||6||Yes|
|3||Students rated as surface learners on the SPQ who responded at a deep level to the test question as rated by the SOLO taxonomy.||9||Yes|
|4||Students rated as surface learners on the SPQ who responded at a surface level to the test question as rated by the SOLO taxonomy.||6||No|
The abbreviations used are: CAL, computer-assisted learning; SPQ, Study Process Questionnaire; SOLO, Structure of Observed Learning Outcomes.