The influence of search stage on gaze behavior in a faceted search interface
This poster reports on a study of gaze behavior within a faceted library catalog. Initial results suggest that gaze behavior differs at definable stages within a search session.
Faceted search interfaces (Tunkelang, 2009) are widely used to support complex and iterative information seeking tasks, such as exploratory search. Recently, libraries have started adopting interfaces to their on-line public access catalogs (OPACs) that include facets such as Library of Congress subject headings and other metadata.
Recent research by our group has suggested that facet use may differ between stages of a search session (Kules et al. 2009). During a search session, searchers move among different stages of the process as their search progresses, such as problem definition, query formulation, results evaluation, and reflection (Marchionini, 1995). An understanding of facet use at different stages will help designers choose facets that better support users' needs.
The preliminary study reported here addresses the question, “Does gaze behavior differ at definable stages within a single search session?” We operationalized this question through two measures: 1) How long do searchers look at the major elements of the interface? 2) How frequently do searchers look at the major elements of the interface?
This study used a mixed factor design to explore the effects of stage of search and three training conditions on interface use. All participants completed a set of six exploratory search tasks (task orders were counterbalanced) and were assigned to one of three training conditions. Stage of search data was collected retrospectively at the end of the study session for the last two tasks that each participant completed. In this poster, we present only results from analysis of the stage of search data. This study extended the procedures and protocols developed in our recent work (Kules & Capra, 2008; Kules et al., 2009).
The exploratory search tasks were situated in the context of preparing to do research for an undergraduate academic paper. For example, one task was:
“Imagine you are taking a class called “Feminism in the United States”. For this class you need to write a research paper on some aspect of the U.S. feminist movement, but have yet to decide on a topic. Use the catalog to find two possible topics for your paper. Then use the catalog to find three books for each topic so that you might make a decision as to which topic to write about.”
Stages of the Search Process
We asked participants to self identify their search stage, so it was important that the stages be meaningful to an undergraduate student. Since most models (e.g., Marchionini, 1995; Järvelin and Ingwersen, 2004) are intended for researchers, we adapted them to produce a set of stages tailored to the tasks and understandable by undergraduate students:
Query Terms – Coming up with search terms
Overview – Getting an overview of the search results
Extracting – Extracting specific information from results
Deciding Next – Deciding what to do next
Deciding Topic – Deciding on a topic
We recruited 18 undergraduate students (8 male, 10 female). They conducted six exploratory searches in a modified version of the North Carolina State University (NCSU) library catalog of over 1.8 million titles (Antelman, Lynema & Pace, 2006). A Tobii T120 eye tracker was used to collect gaze data and screen video of each search. Tobii Studio analysis software was used to segment the page into areas of interest (AOIs) based on interface elements and to compile a data set of gaze data for specific AOIs. This data set was used to compute total fixation times and fixation counts for each AOI. The four AOIs analyzed in this study were the query box, the facets, the breadcrumbs, and the results.
For each participant, after they completed the tasks, their last two searches were replayed at half-speed, with their gaze data overlaid onto the captured screen video. They were instructed to report what they were doing at each point in the video, verbalizing each time they changed stages.
Stage of search data was collected for only the last two searches in order to keep the overall study session time less than 1.5 hours. The study was pilot tested with six participants.
In this poster, we present descriptive data from our preliminary analysis. Table 1 summarizes descriptive statistics about the mean total fixation time in each AOI for each stage of search. The patterns for fixation counts are very similar to the fixation times. Note that each of the search sessions was approximately 6 minutes, but the total of all fixation times (mean of 181 secs) does not include times when their eyes were moving (saccades), looking off-screen, or when the eye tracker was unable to register their gaze location. Although the standard deviations are large, the data suggest interesting patterns. First, searchers spent most of their time (mean of 71 secs, 39% of all fixation time) in the Extracting stage. They spent a mean of 21–35 secs (12–19%) in each of the other stages.
During the Overview and Extracting stages, participants spent about 75% of their time in the results AOI and about 15% in the facets AOI. In the Query Terms stage, they spent about 35% of their time on results and 35% on facets. In the two decision-making stages (Deciding Next and Deciding Topic) they spent about 50–55% of their time on results and 25–35% on facets.
Table 1. Mean total fixation time in each AOI by search stage. Parenthesized values are time spent in the AOI as a percentage of total fixation time for that stage
|Bread crumbs||2.34 (10.1%)||1.96 (5.6%)||3.21 (4.5%)||1.68 (5.4%)||1.68 (7.9%)|
|Facets||.51 (36.6%)||5.87 (16.9%)||11.15 (15.7%)||12.02 (38.3%)||5.88 (27.8%)|
|Query box||4.16 (17.9%)||1.31 (3.8%)||2.61 (3.7%)||1.83 (5.8%)||2.44 (11.5%)|
|Results||8.20 (35.2%)||25.58 (73.6%)||54.23 (76.1%)||15.82 (50.4%)||11.16 (52.7%)|
DISCUSSION AND CONCLUSIONS
The stage of search data suggest interesting behaviors. Not surprisingly, searchers spent much of their time extracting information from the results. Overall, they spent little time looking at the query box, reinforcing its importance primarily for input rather than output. In the Overview and Extracting stages the results were the most used, with about 75% of time and fixations in these stages. However in the Query Terms and decision making stages (Deciding Next and Deciding Topic), facet use is noticeably higher. We hypothesize that during these cognitively demanding stages, searchers may find the additional information provided in the facets and breadcrumbs useful in making decisions.
These results also suggest that while searchers did use facets during the Overview stage, they may be used as much or more in other stages. This does not necessarily mean the facets are not valuable for gaining an overview – previous research has suggested the benefit of this function (Kules & Shneiderman, 2008). It may be that, like the query box, facets are valuable at specific points in the search process.
This research is exploratory – the differences observed need to be analyzed with stronger statistical methods and the data collection methods need to be validated. Although this method and the provided categories inevitably impact the retrospective recall, they provided a direct, fine-grained way to determine search stage. We plan to refine and validate the search stages to ensure that they reflect parts of the search process that are meaningful to searchers. We also plan to refine the protocol for reporting search stages. The half-speed replay with gaze traces provides a way for subjects to promptly report on search activities without interrupting the search process.
We thank Joseph Ryan, Matthew Banta and Abbey Gerken for their assistance. This research was supported by an OCLC/ALISE LIS Research Grant.