Assessing the affective impact of social navigation tools in facilitating exploratory search
Social navigation tools were developed with an aim to guide user exploration of an information space and to inform users' decision making processes (Dieberger, Dourish, Hook, Resnick, & Wexelblat, 2000). In the online bookstore setting, social navigation tools such as book recommendations, user tags and customer reviews address information needs not expressible with keyword search so as to facilitate exploratory activities, which may enhance subjective search experience. In order to examine whether online social navigation tools influence the affective aspects of user experience, theory of flow is applied in this study to form a new evaluation methodology. Impacts of social navigation tools on behavioral variance are also discussed.
Traditionally it is assumed that the most essential task of an IR system is to deliver relevant information; hence the wide adoption of information quality based performance criteria such as precision and recall. Yet with more and more interactive features now made available in online environment, the traditional system-oriented, batch mode retrieval evaluation is ill-equipped to capture all the values or benefits an information service might deliver to its users. Besides the quality of information delivered, more and more attention has been given to the user experience while interacting with the system. This is especially so in the e-commerce context where consumers who purchase online do not merely seek convenient transactions, but also an enjoyable shopping experience (Katerattanakul & Siau, 2001). In this light, information systems utilized in the online commerce setting have to excel in creating pleasurable and fun shopping environments, which points to the need of new evaluation criteria to measure hedonic aspect of user experience.
In the case of online bookstores, chance encounters with items of interest through browsing are now made possible by a variety of social navigation tools, which inform users of what other like-minded people are reading. For instance, users can browse related items by automatic recommendations based on collective purchase decisions, reviews and tags provided by other readers. We postulate that when users are engaged in exploratory search on an online bookstore, the use of social navigation tools would enhance their perceived pleasure of search experience. We propose to measure the hedonic experience using the established instrument derived from the theory of “flow”. Originated from psychological study of the optimal performance of rock climbers and dancers (Csikszentmihalyi, 1988) the theory of “flow” has since been applied to study user experiences in the field of HCI (Novak, Hoffman and Yung, 2000; Ghani and Despnade, 1994; Trevino and Webster, 1992). We conducted an empirical study to determine whether social navigation tools make a difference in users' experience of “flow”, along with the information processing and decision making aspects of user behavior.
RESEARCH DESIGN & PROCEDURES
To compare user experience between interfaces with and without social navigation tools, two mock online bookstore websites were created using Amazon's API. While the control interface provides simple classification-based navigation function, the experimental interface was enhanced with social navigation tools such as automatic recommendations, user tags, and customer reviews (see appendix).
Within-subject experimental design is applied to compare how the perceived pleasure of search varies as subjects shift from the experimental interface to the control interface and vice versa. Every participant was required to complete two search tasks alternately on two interfaces, both of which are open-ended tasks representing exploratory search scenarios. Participants completed an entrance questionnaire prior to the test and an exit questionnaire after each search session. During the 15-minute search session, subjects were asked to select no more than five ideal books based on the assigned search scenarios and save them in a shopping cart. As soon as the search time expired, participants filled out questionnaires adapted from Webster, Trevino and Ryan (1993) which assessed their state of flow, and answered questions related to the degree to which each book they selected might meet their expectation (decision confidence), whether they better understand the options they had for the search scenarios (knowledge gained) and their intention to re-visit the website again in the future.
48 participants were recruited, which resulted in 96 searches with two interfaces. Repeated ANOVA tests were conducted to determine the effect of social navigation tools on the perceived pleasure of search, knowledge gained and decision confidence. Significant difference was found between the two interfaces on the degree of perceived pleasure in terms of flow state (F(1,47)=51.36, p=.000) (See table 1). Tasks searched with experimental interface result in significantly higher flow scores. According to their self-report, the percentage of participants experiencing flow using the experimental interface was also significantly higher than when using the control interface (x2(1)=10.89, p=.001) (See Table 2). It was also found that the experimental interface results in significantly higher degree of decision confidence(F(1,47)=11.68, p=.000), more knowledge gained(F(1,47)=15.04, p=.000) and stronger revisit intention(F(1,47)=38.00, p=.000) (Table 1). Lastly, the transaction logs show that users also were engaged in more exploratory activities such as query submissions and mouse clicks when using the experimental interface.
Table 1. Analysis of assessed degree of flow and behavioral variance
Table 1. Analysis of Self-reported degree of flow
We developed a novel approach for evaluating the affective aspects of exploratory search in the context of online bookstores. The results indicated that users who attempted exploratory search with the guidance of social navigation tools perceived higher degree of pleasure in terms of the state of flow assessed, gained stronger decision confidence and were more likely to return. Further, while search effort were conventionally considered, from usability perspective, as indications of undesirable search costs, it is argued that the results show that the social navigation tools were more conducive to higher level of user online activities. In other words, they promoted higher level of exploratory activities. Our future study will focus on how users' decision confidence and search route vary as they click on different types of social navigation tools. We believe our findings will be useful in improving system design to provide both efficient and enjoyable online exploratory search experiences.
The work was funded by National Science Council (NSC), Taiwan, under grant NSC 98–2410-H-002–226-MY2.