SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Introduction and rationale
  4. Related literature
  5. Research design
  6. Preliminary results
  7. References

People's interaction and negotiation with an information retrieval (IR) system is a constant decision making process, where people make errors as they do in their daily lives. Knowledge of biases and heuristics inherent in human reasoning as suggested by decision-making theories provides us a useful tool to explore the underlying cognitive processes of human errors in using IR systems. In this pilot study, we explore the potential of decision-making theories for explaining human IR errors and propose a tentative knowledge structure of human errors in IR systems.


Introduction and rationale

  1. Top of page
  2. Abstract
  3. Introduction and rationale
  4. Related literature
  5. Research design
  6. Preliminary results
  7. References

Errors are failures of some planned sequence of mental or physical activities to achieve its intended outcome (Reason, 1990). Understanding how and why human errors occur in using IR systems is beneficial. At a practical level, identifying human errors can inform the implementation of future IR systems that are not only easy-to-learn and easy-to-use, but also intelligent in preventing possible errors. At a theoretical level, exploring the nature of human errors can isolate users' misconceptions from their lack of knowledge so that a deeper understanding of the nature of human behavior with IR systems can be achieved (Borgman, 1987).

Human errors with IR systems have been documented widely in information seeking behavior literature. However, the current literature mainly focuses on describing error behaviors without revealing its underlying cognitive mechanisms. In this study, we aim to investigate human errors in using IR systems from a decision-making point of view. The study has two purposes: (1) to introduce the potential of decision making theories in explaining human IR errors; (2) to propose a tentative knowledge structure for classifying human IR errors and developing appropriate remediation strategies.

Related literature

  1. Top of page
  2. Abstract
  3. Introduction and rationale
  4. Related literature
  5. Research design
  6. Preliminary results
  7. References

Decision-making theories and human errors

Psychologists proposed three models of people's thinking: descriptive, prescriptive, and normative models. Normative models define the best thinking for achieving the thinker's goal; that is, if the decision maker conforms to logically compelling properties, the outcome is predictable — an ideal situation. Descriptive models are theories about how people normally think, solve problems, or make decisions. In real life, people often violate normative principles. Thus, there is a gap between the desired normative principles and the observed descriptive models. Prescriptive models are gap-bridging models that “prescribe” or state how we ought to think and provide methods to aid people in conforming to normative models. Because normative models of thinking are often too cognitive demanding to be employed by people in daily decision makings and people's memory and computational capacity are limited, prescriptive models serve as the middle ground between normative and descriptive models. Prescriptive models may consist of lists of useful heuristics, or rules of thumb, much like the heuristics that make up many descriptive models (Keller, 1989; Stanovich, 1999).

Empirical observation of people's decision making found that people's thinking, reasoning, and decision making (descriptive models) constantly deviate from normative models. People systematically make different kinds of judgment and decision errors across problems in different domains. A non-exhaustive list of common errors caused by biases and heuristics includes conjunction fallacy, anchoring, availability, familiarity, based-rate neglect, hindsight, contextualization, over confidence and so on. People's attention also can be manipulated by verbal statements as reflected by framing effect and reference point effect (Baron, 2000).

Studying human IR errors in light of decision-making theories can help us identify the underlying cognitive mechanisms of IR errors.

Knowledge structure of human errors in using IR systems

Borgman (1983, 1986) defined two types of errors when observing OPAC searchers: logical errors (commands that could be partially recognized by the system) and typing errors (commands that could not be recognized at all). Puttapithakporn (1990) classified errors by commercial database searchers into two major types: syntactic and semantic. Syntactic errors are not executed by the system and often evoke error messages. Syntactic errors include incorrect use function keys. Semantic errors, however, are syntactically correct but the outcome does not meet the goal states.

Reason (1990) built an error taxonomy based on a three-level performance framework by Rasmussen (1986): skill-based slips, rule-based mistakes, and knowledge-based mistakes. Skill-based slips are mainly due to monitoring failures; rule-based mistakes arise in the misapplication of good rules; and knowledge-based mistakes are caused by bounded rationality and incomplete or inaccurate knowledge.

The existing literature on knowledge structures mainly focuses on describing and classifying errors in terms of superficial characteristics, such as syntactic/semantic and logical/typing without relating the fundamental mechanisms or causes to the errors. A knowledge structure incorporating decision making theories could bridge this gap by providing explanations for errors at a fundamental cognitive level and therefore could be more effective in guiding system designs.

Research design

  1. Top of page
  2. Abstract
  3. Introduction and rationale
  4. Related literature
  5. Research design
  6. Preliminary results
  7. References

In this research, we analyzed human errors in using IR systems, particularly OPAC, commercial databases, and the Web, reported in published literature. Table 1 shows a snippet of errors identified from the literature. Each type of errors was inspected to see if it can be explained by certain decision-making theories. This approach, as a limitation, does not consider the contextual factors that affect the error behaviors. However, as a study with the main purpose to explore the potential of decision theories to understand underlying cognitive processes of certain error behaviors, it is necessary to omit the context of each study

Table 1. Examples of errors from the literature
LiteratureIdentified errors
Jasen, Spink, and Saracevic (2000)
  • Usage of plus and minus operators (space between + − operators)
  • A carry over from user learning associated with other search engines, including those from other Web, OPAC, and IR systems
Rieh and Xie (2001)
  • Spelling errors
  • Usage of operators (space, +,−,Boolean)
Wang, Berry, and Yang (2003)
  • Adding word(s) or using Boolean operators AND to solve zero-hit problems
  • Using Boolean syntax from other systems
Dickson (1984)
  • Wrong name order, wrong forename, or incorrect inclusion of a middle initial when searching the author name;
  • Exact repeating of previous search

Preliminary results

  1. Top of page
  2. Abstract
  3. Introduction and rationale
  4. Related literature
  5. Research design
  6. Preliminary results
  7. References

Explaining human IR errors using decision-making theories

To illustrate the capacity of decision making theories in explaining human error with IR systems, a few examples are illustrated in Table 2:

Table 2. Examples of decision making theories and their application in explaining human errors with IR systems
Decision making theoryExamples
Dual-Process theory 
The theory argues that human reasoning system consists of two sub-systems. System 1 encompasses primarily the process of interactional intelligence. It is automatic, largely unconscious, and relatively undemanding of computational capacity. System 2 is a controlled processing. It is analytical and requires more computational power (Stanovich, 1999). When the information structure in the environment does not match with the ecological reasoning process of system 1, people tend to make mistakes. In these cases, system 2 should be activated to do analytical thinking and to override system 1 when judgment and decisions need to be made.

The theory explains people's misconception that the size of results increases by adding an AND to search queries. A typical behavior people show in the search process is that: adding word(s) or using Boolean operator AND to resolve zero-hit problems. But in IR systems, AND imposes more restrictions on search conditions and leads to less results.

In daily environment, adding more words and the word AND often connotate the perception of “more”. The conflict between the general meaning of “and”and the “AND” in information retrieval context is a potential cause of this error.

Conjunction fallacy 
The conjunction rule holds that the mathematical probability of a conjoint hypothesis (A&B) cannot exceed that of either of its constituents, that is, p(A&B)<=p(A), p(B). The violation of this “most fundamental qualitative law of probability” (Tversky & Kahnerman, 1983: 294) is called conjunction fallacy.The Boolean logic error described above can be explained by conjunction fallacy as well. The zero-hit result refers to p(A) or p(B). People add A and B together (p(A&B)), hoping to achieve more results. This behavior violate the rule of p(A&B)<p(A), p(B).
Attentional bias 
Attentional bias is a failure to look for evidence against an initial possibility, or a failure to consider alternative possibilities.It is well documented that users often cannot recover quickly from errors; rather they keep trying the strategies that failed them in the first place (Dickson, 1984). Attentional bias can be used to make sense of this phenomenon.
Simple heuristics vs. multi-attribute decisions 
In information searching process, people tend to use simple heuristics, even when a multiple attribute decision making scenario (multi-entry form in advanced search) is provided. In other words, people rarely use multiple cues in decision making.The majority of searches were simple, specifying only one filed or data type to be searched; the advanced search features were rarely used (Mathews, et al., 1983).

Knowledge structure scaffold

Given the important role of the knowledge structure in facilitating understanding and the limitations of the current structures in informing system design, a new knowledge structure of human errors with IR system is needed. We propose a new knowledge structure scaffold with two functions: (1) to help people better understand IR errors; (2) to relate IR errors with system design and user instruction to minimize future errors. A tentative scheme is illustrated in Table 3. Empirical studies are needed to verify the usefulness of the scheme.

Table 3. Knowledge structure scaffold and IR errors in using systems
Types of IR errorsUnderlying causesScaffold
Usability problems

Errors that are caused mainly by interface design.

A typical example is a confusing label of a function in the system. These errors can be reduced or eliminated by improving system interface design or providing effective context-sensitive help.

Comparatively easy to solve and should be addressed with a high priority
Perpetual errors

Errors that are caused by inherent human biases and heuristics, such as the errors outlined in Table 2.

These errors are not necessarily solvable by changing system design, but can be reduced by explicating system's underlying operating process.

Comparatively hard to address, given limited interface space and the perseverance of the biases and heuristics. But this type of errors is comparatively easy to predict
Random errors

Errors that do not occur systematically.

A typical example is typing errors. These errors can be reduced by designing effective user assistant functions (e.g., automatic spelling correction).

Comparatively easy to address; but, because of its idiosyncratic characteristics, this type of errors is hard to predict
Gaps between mental models and system features

This type of errors is caused by mismatch between people's mental models of the system and the system's features or behaviors.

A typical example is that people tend to use natural language when searching in IR systems. This type of error can be most effectively reduced by training or by trial-and-error.

This type of errors is also hard to predict for its idiosyncrasy. But frequently occurred mismatches are comparatively easy to identify

References

  1. Top of page
  2. Abstract
  3. Introduction and rationale
  4. Related literature
  5. Research design
  6. Preliminary results
  7. References
  • Baron, J. (2000). Thinking and Deciding. New York, USA: Cambridge University Press.
  • Borgman, C. L. (1983). End user behavior on an online information retrieval system: A computer monitor study. ACM SIGIR Forum, 17 (4), 162176.
  • Borgman, C. L. (1986). Why are online catalogs hard to use? Lessions learned from information-retrieval studies. Journal of the American Society for Information Science, 37 (6), 387400.
  • Borgman, C. L. (1987). The study of user behavior on information retrieval systems. ACM SIGCUE Outlook, 19 (3–4), 3548.
  • Dickson, J. (1984). An analysis of user errors in searching an online catalog. Cataloging and Classification Quarterly, 4 (3), 1938.
  • Jansen, B. J., Spink, A., & Saracevic, T. (2000). Real life, real users, and real needs: a study and analysis of user queries on the Web. Information Processing and Management, 36, 207227.
  • Keller, L. R. (1989) Decision research with descriptive, normative, and prescriptive purposes-some comments, Annals of Operations Research, 19, (volume on Choice Under Uncertainty edited by Peter Fishburn and Irving H. LaValle), 485487.
  • Matthews, J. R., Lawrence, G. S., & Ferguson, D. K. (1983). Using Online Catalogs: A Nationalwide Survey. New York: Neal-Schuman.
  • Puttapithakporn, S. (1990). Interface design and user problems and errors: A case study of novice searchers. RQ, 30 (2), 195204.
  • Reason, J. (1990). Human Error. Cambridge: Cambridge University Press.
  • Rasmussen, J. (1986). Information Processing and Human-Machine Interaction. Amsterdam: North-Holland.
  • Rieh, S. Y.,& Xie, H. (2001). Patterns and sequence of multiple query reformulations in Web searching: A preliminary study. ASIST Proceeding. Vol. 38.
  • Stanovich, K. E. (1999). Who is Rational? Studies of Individual Differences in Reasoning. New Jersey: Lawrence Erlbaum Associates, Publishers.
  • Tversky, A., & Kahnerman, D. (1983). Extensional versus intuitive reasoning: the conjunction fallacy in probability judgment. Psychological Review, 90, 293315.
  • Wang, P., Berry, M. W., & Yang, Y. (2003). Mining longitudinal web queries: Trends and patterns. Journal of the American Society for Information Science and Technology, 54 (8), 743758.