Extending sense-making models with ideas from cognition and learning theories

Authors


Abstract

In this paper we extend existing sense-making models by adding detail on types of conceptual change and cognitive mechanisms taken from theories of cognition and learning. Our extended model aims to offer a more complete picture of the cognitive processes of sense-making, including the underlying cognitive mechanisms and different types of conceptual changes so that we can link observations to theory. We conducted a preliminary test of the model against think-aloud protocols of users performing intelligence analysis tasks using a multilingual, multimedia information system. Their sense-making processes were consistent with our iterative model of search and sense-making loops. Most of the cognitive mechanisms derived from the literature appeared, with a prevalence of data-driven mechanisms.

Introduction

In this paper we extend existing sense-making models by adding detail on types of conceptual change and cognitive mechanisms taken from theories of cognition and learning. We did a pilot study to see how the model fits the think-aloud protocols from intelligence analysis. The main contribution of this paper is the refined model.

People encounter sense-making tasks every day. Sense-making generally involves the following steps (Pirolli & Card, 2005; Stefik et al., 1999):

  • recognize a knowledge gap;

  • possibly generate an initial structure or model of the knowledge needed to complete the task — concepts, relationships, and hypotheses;

  • search for information;

  • analyze and synthesize information to create an understanding; and possibly

  • create a task product based on this understanding in the form of a report, decision, or problem solution.

A typical example of sense-making is shown in Figure 1:

Figure 1.

A Sense-Making Use Case

Making sense of situations or problems generally involves using information systems. With the advance of information retrieval technologies, standard IR systems support search for pieces of relevant information reasonably well, provided the user can identify her information need or knowledge gap at least to some extent.

However, search is only one part of the total sense-making process. The information found is often fragmented and the relationships are obscure. In order to make use of the information they have found, users need to understand the relationships among the pieces, identify patterns, and build on their previous knowledge to create an updated understanding. Sense-making involves the search for and creation of a structured representation of the task, problem, or domain. Systems should provide support for structure-building. Developing such systems requires a sense-making model that provides enough detail of the sense-making process.

Several descriptive models to capture the processes involved in sense-making have been proposed (Dervin, 1992, 1998; Russell, Stefik, Pirolli, & Card, 1993; Qu & Furnas, 2007; Krizan, 1999; Stefik et al., 1999; Pirolli & Card, 2005). These models provide a fairly high-level description; they do not include detail either of the conceptual changes that occur as the sense-maker's mental representation develops or of the underlying cognitive mechanisms that produce these changes.

Research in cognition, learning, and task-based information seeking and use provides important insights for understanding sense-making. This paper builds on previous sense-making models and applies findings from these areas to propose an integrated sense-making model that provides a more detailed description and examines this model against empirical evidence in think-aloud protocols of intelligence analysis tasks.

Note on terminology. The term “sense-making” has been used with a broad meaning and a narrow meaning. The broad meaning refers to the total process of (1) searching for information and (2) making sense of the information. The narrow meaning is restricted to the processes of relating new information to previous knowledge, creating structures, fitting data into structures to create representations, and thus arriving at an understanding of a situation or phenomenon. Sense-making models are about sense-making in the broad meaning.

We use the term “representation” to include both structure and data organized in a meaningful structure (structure instantiated with data). While others sometimes use it to mean just structure, representations are reflections of users' knowledge of a particular task or problem; they may consist of structural elements (entities, concepts, and/or relationships among them) and data. We use structure as a general term to encompass patterns, schemas, frames, and other terms with similar meaning.

The remainder of the paper is organized as follows:

Section 2 discusses related research, with emphasis on the types of conceptual changes during sense-making and the cognitive mechanisms that produce these changes. It lays the groundwork for our extended sense-making model.

Section 3 presents our sense-making model.

Section 4 discusses the methodology of our pilot study.

Section 5 presents the findings; the model captures users' sense-making processes seen in think-aloud protocols.

Section 6 gives conclusions and implications.

Related Research

Sense-making Models

Sense-making typically involves a series of iterative gap-defining and gap-bridging activities between situations (Dervin, 1992, 1998). Russell et al. (1993) recognized four main processes of sense-making (shown in Figure 2):

  • Search for representation (structure): The sense-maker searches for and creates representations (structures or schemas) that can be used to organize the information needed for the task and puts together her/his own structure based on what she found (generation loop).

  • Create instances of representations: The sense-maker identifies information of interest and encodes it in the representation (data coverage loop).

  • Modify representation: The sense-maker modifies the representations when data is ill-fitted or missing (residue) in the representation (representation shift loop).

  • Consume instantiated representations: The sense-maker consumes the instantiated representation and uses it in performing the task.

Figure 2.

Russell's Sense-making Model (1993)

Figure 3 shows an example of an instantiated structure piece:

Figure 3.

Example of Instantiated Structure

Russell's model indicates the iterative nature of sense-making: processing may go through several iterations until sense-making is successful.

Structural representation plays a crucial role in all sense-making processes. Qu and Furnas (2007) further separate searching for structures from searching for data in the sense-making process. They also try to integrate the two processes and emphasize the bi-directional relationship between search and representation construction.

Krizan (1999) proposed the Cyclical Model of the Intelligence Process, which is a special case of sense-making; this model recognized a series of repeated and interrelated steps:

  • 1.Planning/tasking: Analysts first interpret the task requirements.
  • 2.Collection: Analysts acquire information from various sources, including people and information systems.
  • 3.Processing: Analysts select information based on its plausibility, expectablity, and support for intelligence issues.
  • 4.Analysis: Analysts analyze selected information, give descriptions of the domain, establish explanations of phenomena, and interpret causes and effects.
  • 5.Production: Analysts synthesize all available sources, including the intermediate products of previous steps, to create a comprehensive assessment of an issue.

The product of sense-making — in this case an intelligence report — is disseminated for evaluation and feedback, and the next round of sense-making activities follows.

Through cognitive task analysis, Pirolli and Card (2005) derived a “notional” model of sense-making. They recognized ten processes and six representations (ranging from external raw data to the final task representation) of the sense-making process for intelligence analysis. According to this model, the overall sense-making process consists of

  • information gathering,

  • representation of the information in a schema that aids analysis,

  • the development of insight through the manipulation of this representation, and

  • the creation of some knowledge product or direct action based on the insight.

The production of task output follows the sequence “information →schema →insight →product.” The model further separates two loops of activities:

  • an information foraging loop that involves searching for information, filtering it, and reading and extracting information into some schema;

  • a sense-making loop that involves iterative development of a mental model (a conceptualization) from the schema that best fits the evidence.

This model gives a clear illustration of the steps and outputs involved in a complex sense-making process. However, sense-making does not always have clear beginning and ending points. The simplified “waterfall” model runs counter to empirical evidence about several sense-making tasks -for example, expert decision making (Klein, Moon, & Hoffman, 2006).

To sum up, the existing models reviewed in this section are at a fairly high level of description; in the model we propose in this paper (Section 3) we address:

  • the different types of conceptual changes that happen to the representations during the sense-making process;

  • the cognitive mechanisms that contribute to the creation, modification, and update of the structural representations.

This model provides more detail and supports closer linking to theory by incorporating ideas and concepts from other areas, especially cognition and learning theories.

Learning Theories

Meaningful learning is sense-making, so much can be learned from the literature in learning theory, especially Assimilation Theory (Ausubel, Novak, & Hanesian, 1978), Schema Theory (Anderson, 1984; Rumelhart & Norman, 1981; Rumelhart & Ortony, 1977), and Generative Learning Theory (Grabowski, 1996; Wittrock, 1990).

Through meaningful learning (Ausubel, Novak, & Hanesian, 1978), the learner assimilates new pieces of information into an existing relevant aspect of his/her knowledge structure. Knowledge can be thought of as stored in human memory as schemas with interconnected concepts and relationships, organized in a meaningful way (Anderson, 1984; Rumelhart & Ortony, 1977). To “place” new information into existing knowledge in memory, the learner actively constructs schemas from information found (Grabowski, 1996). When the learner tries to assimilate data that do not fit his/her existing schema, or to integrate schemas that contradict with their existing understandings, she/he feels internal conflict, or cognitive dissonance.

The construction of personalized knowledge structures is the key to sense-making. Research has found that the best task performance on drawing analogies was achieved by learners who work with visual support of a graphical browser and focus on structural relationships (Jonassen & Wang, 1993). Visual tools such as “Advance Organizers” (Ausubel 1978) help learners to capture relationships in the information and bridge gaps between existing knowledge and new knowledge to be learned by visually organizing the structure of a task or problem into which sense-makers can fit facts or data, confirming the need for structure-building tools to support sensemaking.

Conceptual Changes

Several types of conceptual change may occur in the mental representation of knowledge as a sense-maker learns about the task, problem, or situation.

Piaget (1978) recognized two types of conceptual changes in knowledge acquisition:

  • 1.Assimilation: the addition of information to existing knowledge structures, and
  • 2.Accommodation: the modification or change of existing knowledge structures.

Following Piaget, Rumelhart & Norman (1981), Vosniadou & Brewer (1987) have identified various degrees of conceptual change by which existing schemas can be modified by new experience or information; similarly, researchers in education (Chi, 2007) and artificial intelligence (Sowa, 2006) recognized degrees of conceptual change. These notions are summarized in Table 1:

Table 1. Types of Conceptual Change
original image

In this paper, we adopt the categorization of Rumelhart & Norman (1981):

  • Accretion: the gradual addition of factual information within existing schemas.

  • Tuning: the evolutionary conceptual change in the schemas for organizing and interpreting information, including “generalizing or constraining the extent of a schema's applicability, determining its default values, or improving the accuracy of the schema” to better fit the data.

  • Restructuring: conceptual changes that involve the radical change of existing structures or creation of new structures.

These three types of changes in one's mental model of a situation may take place as one acquires and makes sense of new information. The sense-maker updates his or her internal knowledge representations so that the new information can be incorporated into his or her existing knowledge.

Cognitive Mechanisms

Researchers in the areas of reasoning (Arthur, 1994; Johnson-Laird, 1999; Toulmin, Rieke, & Janik, 1979), reading comprehension (Kavale, 1980), and learning (Vosniadou & Brewer, 1987) reported several mechanisms that are important to the processing of information and the accretion, tuning, and restructuring of knowledge. These mechanisms can be applied to sense-making research. They fall into two broad categories:

  • 1.Inductive (data-driven, bottom-up) mechanisms: The sense-maker builds patterns from data and uses the patterns to construct temporary internal models, hypotheses, or schemas to work with (Arthur, 1994).
  • 2.Structure-driven (logic-driven, top-down): The sense-maker uses knowledge schemas and logic to make arguments or reach conclusions.

A sense-maker may use any combination of the bottom-up or top-down mechanisms, depending on the nature of the task, familiarity with the domain, and problem-solving approaches.

A preliminary list of cognitive mechanisms compiled from the literature is incorporated into our extended sense-making model, see Figure 4. Concrete examples for some of the mechanisms can be found in Section 5.

Task-based Information Seeking

Sense-making is often embedded in work tasks. Work tasks are distinguished from information tasks (Ingwersen & Järvelin, 2005), which include both search tasks and sense-making tasks. Work tasks often require at least some degree of sense-making, especially if they show one of the following characteristics (Kim & Soergel, 2005):

  • new situations or problems;

  • complex, less-structured situations or problems;

  • a new domain;

  • an unclear information need.

Most sense-making research involved some type of work task(s). Representations constructed during the sense-making process need to fit the task, or they must be updated (Russell et al., 1993). Research suggests that different types of information (task information, domain information, and problem-solving information) were sought for different types of tasks and/or at different stages of the task; for example, at the beginning (pre-focus) stage, background information is sought (Bystrom, 2002; Vakkari & Hakala, 2000). Kuhlthau (1993/2004) suggested that at the beginning of a task, users use information relevant in general, whereas at the end, they use information more specific to a chosen focus. While working on a task, sense-makers go through different stages and have different foci, which may require different types of information and cognitive mechanisms. We build on these findings to investigate the sense-making stages users go through.

Relevance

Research on topical relevance (Huang & Soergel, 2006) reveals different ways in which a piece of information may be useful to a task, such as providing background information about the task or topic or answering a specific question directly when the task enters the focus stage. This categorization of types of relevance can be used in coding the data on sense-makers processes to better understand the role a piece of information plays in the building of an argument by the sense-maker. Beyond that, the types of relevance could be offered in a sense-making tool to help sense-makers assess how they could use a piece of information in building their argument, and perhaps use this as one criterion in arranging pieces of information.

Extended Sense-making Model

Building on research in sense-making and in cognition and learning, we propose a sense-making model that reflects the iterative nature of sense-making and emphasizes conceptual changes that happen during the process (Figure 4a).

Sense-making involves several processes, which may be executed in many different sequences, depending on the level of existing knowledge and the approach of the sense-maker. The model proposed in this paper tries to capture what is typical or, according to the literature reviewed, most frequent.

Sense-making generally starts with the sense-makers' existing knowledge of a problem or task situation. Sense-makers may start with exploratory search and identify gaps in their existing knowledge, or identify gaps directly by analyzing the problem or planning the task.

Exploratory search is the pre-focus stage of seeking for information. During this process, sense-makers identify a problem, realize they need more information, and learn what information they need to know through exploring or browsing or broad search. Specific foci have not been established at this stage. Exploratory search may be triggered by a (one-time) task, or it could be continuous monitoring or scanning of the environment. During exploratory search, sense-makers may look for both data and structure and move through the structure loop and data loop in an embryonic form.

Focused search is a process in which sense-makers search for information about specific aspects of the task situation, having specific questions in mind. The questions represent the gaps identified through problem analysis and/or through exploratory search.

The identification of gaps happens at various stages with different levels of specificity. At the very beginning, the identified gap is a loose notion of lack of knowledge on some topic or problem. As searching and sense-making continue, more specific gaps may be identified, including data gaps and structure gaps.

If a structure gap is identified, sense-makers may use varying combinations of:

  • searching for structures created and described by others and putting together a structure from what they find combined with what they already know;

  • building their own structure or structure modification by examining the relationships of various parts of the structures in their existing knowledge and looking for patterns in the data.

If a data gap is identified, the sense-maker conducts focused search, looking for the particular pieces of data, and fits the data found into the previously built structure (instantiating representations).

There are two mini-loops involved: the structure loop and the data loop (depending on the focus of a particular iteration of the sense-making process), both of which are embedded in a larger loop of sense-making in which knowledge is consistently updated. Sense-makers may take various paths, and the loops may be closely intertwined.

The existing knowledge representation (instantiated structure) may be updated in all three ways -accretion, tuning, and restructuring - referring to both the processes and outcomes of the change.

Instantiating structures may result in accretion (the data fits with the existing structure) or in tuning (the sense-maker makes minor modifications on the structure to let the data fit). Searching for structure may result in tuning (the gradual change in knowledge structure) or in re-structuring (the radical change in knowledge structure).

Some sense-makers may start top-down, creating structures and then searching for data to fit in. Others may start bottom-up, accumulating (accreting) new data that may lead to tuning, and possibly to complete restructuring.

Several cognitive mechanisms are involved in the processes; Figure 4b gives a preliminary list compiled from the literature. They may be used alone or in combination. For example, a sense-maker may use the “key item extraction” mechanism to extract key entities, concepts, or relationships as the basic structure elements to build on. She may then use “specification” to specify different aspects or requirements of an extracted concept. All mechanisms (data-driven and structure-driven) may be used in instantiating structure (in the data loop) and building structure (in the structure loop) (see Figure 4a).

The ultimate product of successful sense-making is an updated knowledge representation consisting of instantiated structures (or schemas). The mechanisms described above influence the creation of instantiated structures and the knowledge update. Sense-making (or one iteration of sense-making) is completed when the sense-maker incorporates the instantiated structures into her existing knowledge.

Figure 4a.

An Extended Iterative Sense-making Model with Detail on Cognitive Mechanisms (see Figure 4b)

Figure 4b.

Cognitive Mechanisms Used in Sense-making Processes

Methodology

We had available data from participants performing assigned intelligence analysis tasks (information tasks to support someone else's work task) in the context of the formative evaluation of the Rosetta system. This is a news (print, radio and television) retrieval system covering both broadcast and Web-based news in several languages (Arabic, Chinese, English, and Spanish), automatically transcribed and translated into English as necessary (Zhang, et al, 2007). Automatic translation and transcription introduced a number of errors, which made intelligence analysis challenging. We analyzed data from this evaluation to have a first look at whether our model comports with processes that sense-makers use.

Participants

Participants included six information science students as surrogate subjects for intelligence analysts, who were not available for a long-term study. The participants were training to become librarians/information specialists and were comfortable with a range of information gathering and analysis techniques. They were tasked with synthesizing data from a variety of sources, assessing the credibility of information, and evaluating claims based on supporting evidence.

The study consisted of six sessions between March 2007 and May 2007, with two to five participants attending each. Our data consists of seventeen think-aloud protocols.

Data Collection

Participants were instructed to think aloud — to verbalize any thoughts that go through their mind as they use the system to perform the analysis task (Ericsson & Simon, 1998; Nielsen, Clemmensen, & Yssing, 2002). Sessions were logged and participants were interviewed after each session.

Research in cognition (Ericsson & Simon, 1998) has found that when people are thinking aloud their sequences of thoughts are not systematically altered by verbalization. However, when people are asked to explain or describe their thinking to another individual, their thoughts are often prone to reactive influences. To ensure that participants have minimal reactive influence on their thinking, we explicitly instructed them to focus on the task while thinking aloud and merely to verbalize their thoughts, rather than to describe or explain them to anyone else. The participants were also given a training task, preparing them for the think-aloud exercise and getting them familiar with the system.

The think-aloud protocols were digitally recorded using a program (Audacity), and transcribed immediately after each session.

Task Scenarios

We used task scenarios that simulate intelligence analysis tasks, information tasks that stem from decision-making tasks in making or implementing policies. A scenario consisted of three sections: information need, output format, and context. Details of task scenario design and testing can be found in Zhang et al. (2007). Figure 5 shows two sample task scenarios:

Figure 5.

Two Sample Task Scenarios

Data Analysis

We conducted a preliminary test of the refined model against the participants' sense-making processes primarily based on the think-aloud protocols. The unit of analysis, or case, is one participant performing one task. A single coder (the first author) coded the transcripts, using the initial coding scheme shown in Table 2. This coding scheme was derived from the literature as reviewed in Section 2. When there was ambiguity in deciding the code for a think-aloud piece, the coder checked additional data sources, including query logs and interview notes.

Search and sense-making processes, conceptual changes to existing knowledge, and mechanisms that enable the processes and changes were recognized. Based on the coding, we further divided the protocols into process loops: each loop consisted of some search processes followed by some sense-making processes. We then divided the entire protocol into several stages reported in Section 5.

Table 3 shows a sample section of a think-aloud protocol with coding.

Table 2. Coding Scheme
original image
Table 3. Sample Think-aloud Protocol Section with Coding
original image

Limitations

This is a pilot study that was done with a particular user group (information science students as surrogate subjects for intelligence analysts) in a particular domain (intelligence). It is not a comprehensive test of our general model. The participants were working with a collection that contained translated materials. We observed cases where sense-making failures were due to translation errors. The processes that participants went through and the cognitive mechanisms that they used thus might be influenced by the quality of the translations.

Findings

Search and Sense-making Loops

The think-aloud protocol analysis shows that each participant's overall sense-making process consisted of several loops, each having search processes followed by sense-making processes. A basic pattern of moving through the iteration of loops emerged, but there were variations.

In most cases, the overall sense-making process followed four stages, with variations in length, number of loops involved, and focus:

  • 1.Task analysis: The participant started with a task analysis stage, in which they established their initial representation of the task situation, requirements, and sometimes their strategies to accomplish the task.
  • 2.Exploratory stage: The participant searched for background or general information about key figures or places in the task scenario. Sense-making in the exploratory stage often involves identifying several knowledge gaps and extracting entities or concepts that are related to the key concepts mentioned in the task scenario or their previous knowledge. An initial representation was established.
  • 3.Focused stage: The participant searched for various specific aspects of a topic to answer specific questions. Sense-making in the focused stage involves raising specific questions, filling in a great deal of factual information, and sometimes tuning or restructuring the structures, which result in a finer or more detailed representation.
  • 4.Updates of knowledge representation: Often the updates of representation were embedded in the sense-making processes, but sometimes the participant explicitly iterated the process to update his/her understanding.

In general, search and sense-making loops move from exploratory, to focused, to more focused. In the exploratory phase, participants often mentioned that they were searching for “general” or “background” information to “get an idea of what's going on there.”

Some tasks included several sub-tasks, each requiring its own sense-making process sometimes involving all four stages.

Sometimes, if the sub-tasks or concepts were similar enough, participants skipped the exploratory stage for the new concept, adopting instead the conceptual schema developed during the exploratory phase for other similar concepts. This pattern was evidenced by all participants for one task which involved collecting information about several oil-exporting countries and assessing their political and economic stability. When participants found certain issues in one country through the exploratory phase, they directly searched for those concepts in countries having similar geographic or political environments without going through an exploratory phase.

We also examined reasons for starting a new loop of search and sense-making, which included:

  • Success of previous sense-making: Participants successfully updated their knowledge about the concept or relationship and moved on to the next concept or relationship.

  • Failure of previous sense-making: Participants failed to make sense of the search results and gave up on this particular concept or relationship. Failure may be caused by fragmented information, conflicts with existing knowledge, and/or translation errors.

  • New lead: Through exploratory search and sense-making, participants discovered new concepts or relationships that needed further investigation or suggested new search terms for existing concepts.

  • Failure of search: Participants could not find any or not enough results on this particular concept or relationship, or they found too many.

There were two patterns through which foci were established during the exploratory phase:

  • 1.Data driven: narrowing down because of too much data.
  • 2.Structure-driven: discovering new conceptual aspects through analysis.

Data- and structure-driven approaches were present throughout the sense-making processes at various stages.

Data- vs. Structure-Driven Approaches

Participants used a two-way approach: data-driven (bottom-up) and structure-driven (top-down) with different degrees of emphasis. When the two directions met, sense-making was successful.

As to the cognitive mechanisms involved, most participants used data-driven mechanisms about four times more often than structure-driven mechanisms. This may be due to our participants' relative inexperience with the task domain (intelligence) and to the need to overcome the spotty data problem from imperfect text. The average number of occurrences of different cognitive mechanisms for each task (T1 through T6) is shown in Table 4:

Table 4. Occurrences of Cognitive Mechanisms
 TIT2T3T4T5T6AVG
Data-driven1726159131817
    Key item extraction12139771110
    Comparison3441233
    Similarity1211132
    Differentiation12  111
    Analogy/metaphor 3  1 1
    Classification       
    Schema induction 11    
    Generalization 1  1  
Structure-driven3416224
    Definition       
    Specification 112 11
    Explanation-based22 11 1
    Elimination11 1  1
    Semantic fit       
    Socratic dialogues       
    Inference   2111

These cognitive mechanisms appeared at several stages. For example, key item extraction was present throughout the sense-making process; it was the most used mechanism for all tasks. The extracted items may be entities, concepts, or relationships. Participants scanned for key items or names as a basis for deciding whether to read the full text. They first extracted key concepts and then used other mechanisms, such as comparison, to relate them to concepts they had previously found or to their existing knowledge. Extracted entities, concepts, and relationships also provided them with new search leads.

Through comparison and analogy, participants were able to adapt a structural representation of one concept to another similar concept. “…with all the violence in Nigeria, I was expecting to find the same types of political outrage in Algeria…”

For other similar concepts, they also used query structures and query terms similar to those they had found successful through several rounds of exploration. For example, in the sample protocol section, one participant used the same query for Algeria, since the query was very successful for another country, Nigeria, which the participant believed to be “close”: “…that was actually a very useful search. So let's still take this query and look at Algeria, because obviously Algeria and Nigeria are very close…”

As for whether the data-driven or the structure-driven approach was more helpful, that seemed to depend on the nature of the task, the characteristics of the participant, and the information that was indexed by the search system; this needs further investigation.

Conceptual Changes to Knowledge. Dealing with Conflicts

When participants detected conflicts, either between two pieces of information they found or between new information and their existing knowledge, we observed four ways in which the conflicts were dealt with (Plous, 1993):

  • 1.Disregard: The participant refused to accept conflicting evidence and kept the original conceptual representation; no conceptual changes happened. “… I wanted that article to say something else. I have to disregard it.”
  • 2.Compromise: The participant partially accepted the conflicting evidence and partially changed his or her existing conceptual model to integrate the new evidence. This was when tuning took place. “…it is not my understanding that this has anything to do with oil. But okay, this is a different take.”
  • 3.Acceptance: The participant updated his/her existing representation and accepted the new evidence. As a result, the existing conceptual model often had to be restructured. “…I thought these countries had similar serious problems. But [in fact, they] seem to be relatively stable and put a lot of effort into establishing their economy.”
  • 4.Confusion: The participant failed to resolve the conflicts. “…Obviously I have no idea what this is about…” This often resulted in an unstable or unsatisfied mental state. Sometimes the participant searched for more information until the confusion was transformed by new evidence into one of the above three states. Sometimes a participant simply gave up and moved on.

The changes might be influenced by the strength and coherence of the new information and the strength, coherence, and degree of commitment to the users' previous knowledge (Dole & Sinatra, 1998).

It was not possible to observe all the conceptual changes happening to participants based on the think-aloud data alone, especially when the information fit with the participants' existing knowledge. When conflicts happened, participants often explicitly talked about how they dealt with the conflicts and the results of that conflict. But when data fit (the task reports included many pieces of data that were not commented upon in the think-aloud protocols), they did not always explicitly acknowledge the acceptance (or disregard) of a piece of information. In the think-aloud transcripts we observed a large amount of factual information being processed by the participants. However, the think-aloud protocols did not reveal whether the factual information resulted in less radical changes (accretion or tuning) in the participants' knowledge schema, with major restructuring having a better chance of being captured. Tuning and re-structuring could be observed when conflicts were encountered and the participants explicitly talked about how they dealt with them. However, the textual representation of participants' thoughts did not convey a complete picture of the previous and changed knowledge states.

In future studies we will provide tools for explicit structure representation, such as concept maps and templates, and keep a detailed log of users' intermediate structures, notes, and drafts to shed light on this question.

Role of Instantiated Structures

The extraction of entities and relationships (including relationships within the new information and relationships of the new information to existing knowledge) were found to be very important for sense-making.

  • Entities (represented as names) and key concepts, (represented as keywords) were often the basis for relevance judgments. Participants decided to look at the full-text document if the entities or key concepts in a brief summary seemed to be related to their task.

  • The relationships embedded in new information and between the new information and participants' previous knowledge seemed to play an important role in sense-making (structure building and data fitting).

  • Both concepts and relationships seemed to be crucial for updating knowledge. Comparison and dealing with conflicts happened at the level of concepts or overall structure, not at the level of individual data items.

Conclusions and Implications

Conclusions

In this initial study, the think-aloud protocols of student participants performing intelligence analysis tasks appear to be consistent with our iterative model of successive search and sense-making loops. The exploratory and focused iterations took place in many different sequences, depending on the task, the level of existing knowledge, the approach of the participant, and the nature of the data.

The data loop and the structure loop were closely related. It was very rare that a sense-making loop succeeded by searching for and making sense of either only data or only structure alone. In most cases, the sense-making loop had a focus — either data or structure, but it also involved the other. For example, a structural loop may result in searching for data to instantiate that structure, or may result in re-structuring if the data is not compatible with the structure. On the other hand, the data loop may result in the abstraction of structure elements which then lead to tuning or restructuring of structure. In exploratory loops, the focus on data or structure was not very clear; when the sense-maker entered a focused stage, the data loop and the structure loop were more easily separated. Exploratory iterations may be seen as intertwined data and structure loops, while focused iterations can be described as data loops and structure loops following each other.

Participants used mostly data-driven mechanisms for these tasks, which may have resulted from gaps in their domain knowledge that would not have been as prevalent if they were actual intelligence analysts with grater background knowledge. The key item extraction and comparison mechanisms were extensively used by participants for all tasks.

The think-aloud protocols allowed us to track the sense-making processes and mechanisms that participants engaged in. They also point out the need for detailed logging to capture conceptual changes that are not always explicitly revealed in the think aloud protocols.

Implications and Future Work

Theoretical implications. The model may lead to a better understanding of sense-making processes by extending the existing sense-making models to theories in cognition and learning and linking the extended model to observations in sense-making tasks.

Design implications. We believe that the detail in our model provide a better basis for designing sense-making support systems. For example, the extensive use of cognitive mechanisms such as “key item extraction” and “comparison” highlights the importance of extraction of entities, concepts, and relationships, which suggests that automatic information extraction might facilitate some sense-making tasks; this merits further investigation.

Stefik et al. (1999) suggested that when a sense-making task is difficult, sense-makers use external representations to store information for repeated manipulation and visualization. Our model provides the basis for designing and evaluating tools that help structure such representations in a sense-maker's conceptual space to provide better sense-making support to information system users.

The model proposed in this paper deals with individual sense-making, but much sense-making activity occurs in groups. Future work includes expanding the model to collaborative sense-making and investigating how system tools may assist collaborative sense-making. Tools that enable systematic note-taking, well-structured external representation of knowledge schemas may be even more important for people working together in sense-making.

Acknowledgements

This research has been supported in part by DARPA contract HR-0011-06-2-0001 (GALE). We thank Yan Qu for helpful comments on a draft of this paper.

Ancillary