Fishing in query pools for task representations

Authors


Introduction

There is a close relationship between queries and search tasks. Queries are (partial) representations of a searcher's interpretation of a given task topic, and they are also tools constructed by the searcher to carry out the task. In general, search systems treat successive queries entered in a single search session as though each was independent of the previous one, and yet, for complex tasks, the set of queries employed in a single session can be viewed as an expression of the interactive search process. As such, these query chains may reveal search strategies and behaviours that are task dependent.

In most query research, queries are measured according to length (Jansen et al, 1998; Toms & Freund, 2003; Lau & Goh, 2006); counted by task (Toms & Freund, 2006); linguistically analyzed (Toms & Freund, 2003), and grammatically/syntactically checked (Spink & Gunar, 2001; Toms & Freund, 2003). Relatively little work has been done to analyze web query sets or chains, but what has been done suggests that patterns exist which offer potential to distinguish between different search task types (Navarro, Scaife & Rogers, 1999) and predict search strategies and preferences (Radlinski & Joachims, 2005). As the interest in contextual retrieval and implicit measures of search behaviour continues to grow, it is important to consider what we can learn about the tasks in searchers minds from the queries they employ. In this research, we hypothesized that a relationship exists between tasks and query sets, such that the query set will reflect differences in task types and characteristics.

To test this hypothesis, we analyzed query pools - sets of queries collected from groups of users performing the same search tasks - and constructed concept maps showing the core elements and relationships within each pool. Queries are the user's interpretation of the task and from the user's communication with the system. In general, a search system in response to a query treats successive queries entered in a single search session as though each was independent of the previous one. Yet in complex problems, the only productive means of searching for information is to decompose a problem into a series of queries.

In most query research, queries are measured according to length (Jansen et al, 1998; Toms & Freund, 2003; Lau & Goh, 2006); counted by task (Toms & Freund, 2006); linguistically analyzed (Toms & Freund, 2003), and grammatically/syntactically checked (Spink & Gunar, 2001; Toms & Freund, 2003). To date, this has resulted in quantitative profiles and descriptions of queries, with very little evidence concerning the relationship between the query and other contextual factors within the user's environment.

In this research, we hypothesized that a relationship exists between task and the query pool, and that a query pool may be used to re-engineer a task, and additionally infer certain characteristics of that task. To respond to this objective, we mapped the query contents collected from a group of users to concept maps that reflect the core elements that are represented among the set.

Methods

The data used for this analysis was collected as part of a larger study in which 96 people searched three tasks from a set of 12, each of which was identified as a particular type: Decision Making (select a course of action from among alternatives), Fact Finding (find specific facts), or Information Gathering (collect multiple pieces of information about a topic). Each task was also formulated with a particular structure: Parallel (multiple concepts are present at the same conceptual level) or Hierarchical (single concept with multiple attributes).

The participants searched a version of Wikipedia using a Lucene toolkit implementation and a customized interface. Over the course of the study over 1000 queries were generated. All duplicate queries were removed, and the remaining queries were used for this analysis. Three research assistants created concept maps for each task. A concept map is a visualization of the concepts, core elements, and the relationships among them. In this case all of the unique words and phrases were ‘interconnected’ based on their relationship to each other. The type and structure variables were unknown to the classifiers.

Results

Overview

All resulting concept maps had one or more core concepts that were central to a cluster that also contained a series of sub-concepts. In many cases, a map contained multiple ‘core’ concepts, as participants selected terms that were broader than the intent of the task, e.g., art for impressionism, and civil engineering for bridges. Many types of ‘qualifiers’ were used such as instances or names of the concept, types, effects and so on.

By Structure

Hierarchical tasks had a linear structure that started at a high conceptual level such as civil engineering, and moved in increasing specificity to lower level concepts, e.g., bridges that were qualified by one or more sub-concepts such as location (e.g., frozen water), types (e.g., suspension, retractable) and attributes (e.g., 1000 m). Parallel tasks had a broader structure, with multiple concepts on the same level (e.g., castle and fortress), each of which contained identical clusters that included elements such as names and definition. Notably queries for parallel tasks were more likely also to include “versus” or “vs” as a keyword.

By Type

Concept maps of Fact Finding tasks were the simplest, often a single simple cluster. The queries had little deviation in language, terms or concepts.

Information Gathering tasks resulted in concept maps that were the most “sprawling,” containing more unique concepts than the other two. For example, Q9 referenced the book “Fast Food Nation” and mentioned food additives; the resulting queries included a range of terms such as “genetically modified foods”, specific ingredients (e.g. MSG, dimethyl dicarbonate), concepts for levels of government (Canadian Food standards, Canadian government, CFIA), as well as a specific journal name (European Journal of Nutrition).

Query concept maps for Decision-making tasks contained more comparison words and concepts, and more qualifiers and attributes. For example, a task about transportation contained many concepts for type of transportation (car, rail, eurorail, tgv), and a task concerning tourism in France contained multiple layers of requests for maps and different types of tourist information. The concepts maps overall appeared “deeper” and to include judgment concepts (quality, impact).

Conclusions

Although we were working with a limited number of tasks in each group, the concept maps showed relatively unique ‘thumbprints’ for each category of the task variables. Our analysis included all queries without distinction by individual, and did not examine query reformulation (the temporal relations among queries). However, we speculate that sessional query chain sets (or ordered sets) per person could be treated as an incremental and growing set with the potential for a search engine to ‘learn’ from previous behavior and take into account previous query submissions somewhat similar to Radlinski and Joachims (2005) who examined pairs of queries. Although we did not take order of query creation into account in this analysis, one could speculate on whether the chronological set The accompanying poster visually represents this work including examples of the tasks used as well as the concept maps that portray each task.

Ancillary