A platform, in computing terms, is capable of supporting multiple types of systems. Consequently, we are exploring two distinct models of educational software using CogSketch: Sketch Worksheets and the Design Buddy. Each is discussed in turn.
5.1. Sketch worksheets
Paper-based worksheets are a staple in many classrooms. For example, in a geology class, students might be asked to highlight a fault on a photograph or draw the layers of the Earth. Sketching is a valuable way of learning spatial relationships. With pencil-and-paper sketching, feedback is delayed, and assessment is time-consuming and difficult. Experience with intelligent tutoring systems indicates that immediate feedback leads to better student learning (Corbett & Anderson, 2001). With CogSketch, we can provide rapid feedback to students, and hopefully make assessment simpler and more efficient, thereby improving learning.
Fig. 6 shows a sketch worksheet for a physical geology class, where the ink illustrates a typical student response. Students outline the geological features by creating glyphs, labelled with the appropriate concept, over the photograph. Coaching is provided by using SME to compare the student’s sketch with the instructor’s sketch. (Internally, the student’s sketch is a subsketch, and the instructor’s sketch is another subsketch which is kept hidden from the student.) Potential problems with the student sketch are found by analyzing the correspondences and candidate inferences of the mapping that SME produces for this comparison. For example, when a worksheet is developed, the instructor marks which facts are important and what advice to provide if they are not in the student’s sketch (e.g., “Is this really the location of the hanging wall?”), as indicated by a candidate inference from the teacher’s sketch to the student sketch involving that fact. The student can then move and/or redraw their glyphs to improve their sketch, and ask for more help. Worksheets are developed through an authoring environment provided by CogSketch. Authors choose what concepts to include as possible conceptual labels, whether numerical values should be entered for some properties (e.g., the radius of the Earth’s core, in a worksheet on the layers of the Earth) via annotation glyphs, and names and commentary on each concept and relationship, if the defaults are not suitable.
Figure 6. A CogSketch worksheet with student response. Students were asked to draw the fault, the marker beds and which way they moved, and identify the hanging wall and foot wall.
Download figure to PowerPoint
The sketch worksheet model is designed to be simple and general-purpose: If the appropriate KB concepts (or close stand-ins) can be found, a worksheet for that problem can be created. The current version of the authoring environment requires some understanding of the OpenCyc KB conventions and contents, which is daunting for most instructors without help. Most ITS authoring environments do not use independently developed off-the-shelf knowledge bases (e.g., Aleven, Sewall, McLaren, & Koedinger, 2006), requiring them to start from scratch (or from previously constructed systems with the same environment), which is more work but does ensure that whatever linguistic information the environment needs is entered along with new knowledge. We plan to use and expand natural language resources in the KB to make authoring easier. The first classroom use of worksheets was Fall 2009, in a physical geology course at Northwestern University taught by Prof. Brad Sageman. The first assignment involved four sketch worksheets and was quite successful, based on instructor and student feedback. Consequently, the instructor added a second sketch worksheet assignment, where students drew the constituents of the carbon cycle, was added (Yin, Forbus, Usher, Sageman, & Jee, 2010).
There is already some evidence from a laboratory experiment that CogSketch could be useful in automated assessments. Jee, Gentner, Forbus, Sageman, and Uttal (2009) found that when experts versus novices drew geological processes, or marked up images with geological features, there were distinct and easily recognizable differences between the two groups. In process diagrams, experts tended to include more arrows, which in such diagrams indicate the processes that are occurring and relate different aspects of the cycle to one another, and they tend to begin their diagram with such information. In marking up photographs of geological formations, the experts tend to draw more geologically relevant features, often in an idealized manner. This cannot be attributed to differences in drawing skill, since drawings of control photographs (e.g., fruit, lasagna) were indistinguishable. Importantly, the same pattern of results hold for sketching from memory, for copying, and for tracing. This suggests that comparisons of student sketches in a very simple copying task could be diagnostic of their mental models of the domain, analogous to the use of a sorting task to ascertain expertise (Chi, Feltovich, & Glaser, 1981). As other researchers have noted (Cheng & Rojas-Anaya, 2008), timing information can provide another implicit measure of expertise. CogSketch records timestamps for each ink point drawn and other interface events, which we are using to investigate other possible assessment measures.
5.2. Design Buddy
Engineers must communicate with their teammates and with clients in developing and refining designs. Using sketches to communicate is an essential part of the process. CAD software is only used in later stages of design, called detailed design. The early stages (conceptual design) are where the key ideas are worked out, to see if a design might be suitable before doing detailed designs. At Northwestern, first- and second-year students learn design and communications in an integrated manner, creating designs and prototypes that address real-world problems for external clients. Examples include patients at the Rehabilitation Institute of Chicago, whose physical handicaps require new tools to help them accomplish everyday tasks, such as chopping vegetables or trimming their nails. The instructors find that one of their hardest pedagogical problems is teaching students to use sketches to communicate their designs. Consequently, we are creating a CogSketch application, the Design Buddy, to tackle this problem.
The Design Buddy is a form of teachable agent (Blair, Schwartz, Biswas, & Leelawong, 2006) that gives students practice in explaining designs. Students explain their design by drawing a set of subsketches indicating the distinct intended behaviors of their design (see Fig. 2). Transitions on the metalayer indicate how one behavior leads to another. This is an example of a comic graph, which can be viewed as a form of comic strip, although there can be branches (representing different possible outcomes) and cycles (representing repetitive behaviors). In addition, they can make certain kinds of simple English statements about particular states, transitions between them, and purposes using a form-based interface. The Design Buddy critiques their description of intended behaviors, providing feedback to the student. It does this by qualitatively reasoning about the behaviors it believes are possible in the system as sketched and comparing them to the behaviors described by the student to look for mismatches. It also looks at each transition in the student’s sketch, analyzing it to see if it is physically possible, given what it knows. Discrepancies between the student’s explanation of the intended behavior and the Design Buddy’s understanding of the possible behaviors are used to provide feedback (Wetzel & Forbus, 2009).
This application is significantly more difficult than worksheets for four reasons. First, it involves more substantial domain reasoning, rather than just matching. Design Buddy uses a qualitative model of mechanics (Kim, 1993; Nielsen, 1988) for causal reasoning about surfaces, forces, and motion. Qualitative mechanics is a natural fit for conceptual design: Most of the parameters needed for numerical simulation simply do not exist at this stage of design, and qualitative models capture the kinds of causal explanations that designers produce when talking about their designs. Second, the interface must be sufficiently natural to communicate complex behaviors without distracting the student too much. Third, we must develop coaching strategies that help students learn to explain designs in terms that practicing engineers would use. Finally, the assigned projects change every quarter, and a wide variety of design problems arise. We have started to tackle the last problem, of ensuring broad coverage, by analyzing a corpus of student designs from previous years. Out of 39 projects, 19 did not involve mechanics or motion of solid objects (e.g., electrical circuit problems, fluid flow problems), and hence CogSketch could not handle them. Of the remaining 20, four required 3D reasoning beyond what CogSketch can currently do, four required reasoning about gears, but the final 12 designs could be handled by CogSketch in its current form. Currently we are extending CogSketch to handle all of the motion-oriented designs, and doing pull-out studies with Northwestern students to refine the interface and coaching strategies.