SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Background
  5. Method
  6. Analysis
  7. Design Implications
  8. Conclusion
  9. Acknowledgments
  10. References

Support for explicit collaboration in information-seeking activities is increasingly recognized as a desideratum for search systems. Several tools have emerged recently that help groups of people with the same information-seeking goals to work together. Many issues for these collaborative information-seeking (CIS) environments remain understudied. The authors identified awareness as one of these issues in CIS, and they presented a user study that involved 42 pairs of participants, who worked in collaboration over 2 sessions with 3 instances of the authors' CIS system for exploratory search. They showed that while having awareness of personal actions and history is important for exploratory search tasks spanning multiple sessions, support for group awareness is even more significant for effective collaboration. In addition, they showed that support for such group awareness can be provided without compromising usability or introducing additional load on the users.


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Background
  5. Method
  6. Analysis
  7. Design Implications
  8. Conclusion
  9. Acknowledgments
  10. References

Information seeking is typically considered a solo activity, but there is evidence to suggest otherwise. For instance, Twidale and Nichols (1996) pointed out a problem: “The use of library resources is often stereotyped as a solitary activity, with hardly any mention in the substantial library science and information retrieval literature of the social aspects of information systems.” They argued that introducing support for collaboration into information retrieval systems would help users to learn and use the systems more effectively. Similarly, based on their extensive study of patent office workers, Hansen and Järvelin (2005) also concluded that the assumption that information retrieval performance is purely individual should be reconsidered. Morris (2007) proposed that four features of exploratory search experience—coverage, confidence, exposure, and productivity—could be enhanced by providing explicit support for collaborative search and subsequent sensemaking processes.

These demonstrations of collaborative information seeking (CIS) as a phenomenon inspired a variety of systems that aim to support collaborative information seeking. Systems such as Ariadne (Twidale, Nichols, & Paice, 1997) and SearchTogether (Morris and Horvitz, 2007) provide user interface-level support for collaboration. Other approaches (e.g., Pickens, Golovchinsky, Shah, Qvarfordt, & Back, 2008) have focused on providing system-mediated collaborative support. Despite these encouraging efforts, there is a significant need to address a number of CIS-related issues by understanding and addressing the users' needs in a collaborative environment. Grudin (1994, p. 93) noted: “Many expensive failures in developing and marketing software that is designed to support groups are not due to technical problems. They result from not understanding the unique demands this class of software imposes on developers and users.” One such demand or requirement for CIS systems is the support for awareness (Schmidt, 2002, p. 285). During our participatory design sessions,11 we also discovered the importance of providing appropriate awareness functionality on a CIS interface. This led us to investigate the impact, cost, and implementation related issues of awareness support in user-centric CIS systems.

To facilitate this investigation, we have developed a CIS system, Coagmento,22 which provides support for communication and awareness of personal and group histories while working on CIS tasks. Using Coagmento, we conducted a user study that involved 42 pairs of participants, who worked on exploratory search tasks. We assigned each pair randomly to three different versions of Coagmento, where the independent variable was the level of awareness support provided. The participants were brought to the lab for two different sessions, thus allowing them to get used to the system and for studying multisession collaboration. In the present article, we describe this user study, showing how providing different levels of awareness support to collaborators working on exploratory search tasks can impact their productivity, physical and cognitive efforts, satisfaction, and engagement.

The rest of the article is organized as follows. We first present a brief overview of related works and place our research in context in the next section. We then describe our methodology, explaining different aspects of our user study. The analysis and discussion are followed by design implications and concluding remarks.

Background

  1. Top of page
  2. Abstract
  3. Introduction
  4. Background
  5. Method
  6. Analysis
  7. Design Implications
  8. Conclusion
  9. Acknowledgments
  10. References

Three components specific to group-work or collaboration that are highly predominant in the CIS or Computer-supported Cooperative Work (CSCW) literature are control, communication, and awareness (Rodden, 1991).

Several systems supporting collaboration have identified the above-mentioned issues (control, communication, and awareness) as critical to their design and associated user response. For instance, Farooq, Ganoe, Carroll, and Giles. (2009) presented a collaborative design for CiteCeer, a search engine and digital library of research literature in the computer and information science disciplines. Based on a survey and follow-up interviews with CiteSeer users, the authors presented four novel implications for designing the CiteSeer collaboratory: (a) visualize query-based social networks to identify scholarly communities of interest, (b) provide an online collaborative tool support for upstream stages of scientific collaboration, (c) support activity awareness for staying cognizant of online scientific activities, and (d) use notification systems to convey scientific activity awareness.

Awareness is one of the most important issues that is identified and addressed in the CSCW literature. A clear definition and a methodology for providing awareness, though, are lacking. One of the often-asked questions about awareness in CSCW is “awareness of what?” Schmidt (2002, p. 288) argued that we should talk about awareness not as a separate entity, but as someone's being aware of some particular occurrence. In other words, the term “awareness” is meaningful only if it refers to a person's awareness of something. Heath, Svensson, Hindmarsh, Luff, and Lehn (2002) suggested that awareness is not simply a “state of mind” or a “cognitive ability,” but rather a feature of practical action that is systematically accomplished within developing course of everyday activities.

Several related terms and definitions are used in the CSCW literature to refer to awareness in collaborative projects. For instance, Dourish and Bellotti (1992, p. 107) defined awareness as “an understanding of the activities of others, which provides a context for your own activity.” Dourish and Bly (1992, p. 541) suggested the following definition for awareness: “Awareness involves knowing who is ‘around,’ what activities are occurring, who is talking with whom; it provides a view of one another in the daily work environments. Awareness may lead to informal interactions, spontaneous connections, and the development of shared cultures—all important aspects of maintaining working relationships which are denied to groups distributed across multiple sites.”

A set of theories and models for understanding and providing awareness emerged in the early works reported in the CSCW literature. Gaver (1991) argued that an intense sharing of awareness characterizes focused collaboration in which people work closely together on a shared goal. He further claimed that less awareness is needed for division of labor, and that more casual awareness can head to serendipitous communication, which can turn into collaboration. He proposed a general awareness model that incorporates and supports all of such activities.

Bly, Harrison, and Irwin (1993, p. 29) also identified the importance of such general awareness by saying, “When groups are geographically distributed, it is particularly important not to neglect the need for informal interactions, spontaneous conversations, and even general awareness of people and events at other sites.”

Some of the early works, which reported using ethnographic field studies in CSCW (e.g., Harper, Hughes, & Shapiro, 1989b; Harper, Hughes, & Shapiro, 1989a; Heath & Luff, 1991 not cited), identified the need to seamlessly align and integrate the activities of the participants of a collaborative project. Although they did not refer to it as awareness, soon, the term awareness was adopted to address such practices that support connecting collaborators without the activities of asking, suggesting, requesting, ordering, or reminding.

Several works argued that providing audio-video communication channel could suffice for awareness (Mantei, Baecker, Sellen, Buxton, & Milligan, 1991; Gaver et al., 1992). However, the use of communication as a substitute for awareness turned out to be very limited in its applicability (Gaver, 1992). Another line of research focused on providing awareness using computational environments based on “event propagation mechanisms” for collecting, disseminating, and integrating information concerning collaborative activities.

To summarize, there are several ways of defining and implementing awareness. Various research projects have used their own taxonomy and interpretation of awareness for creating frameworks and systems. For instance, Gutwin and Greenberg (2002) classified awareness in two types: situational, and workspace, and they suggested that situational awareness underlies the idea of workspace awareness in groupware systems. Their definition of workspace awareness included how people interact with the workspace, rather than just awareness of the workspace itself. Simone and Bandini (2002) identified two kinds of awareness: (a) by-product awareness that is generated in the course of the activities people must do to accomplish their collaborative tasks; (b) and add-on awareness that is the outcome of an additional activity, which is a cost for the collaborators to what they must do and is discretional in that it depends on collaborators' evaluation of the contingent situation. Chalmers (2002), likewise, divided the awareness in two kinds: awareness of people and of information artifacts. He suggested implementing activity-centered awareness tool, in that it focuses on presenting the ongoing appearance and activity of people.

For the purpose of the work reported here, a more comprehensive and well-accepted taxonomy of awareness, which addresses four kinds of awareness (Liechti & Sumi, 2002) as listed below, will be used.

  • Group awareness. This kind of awareness includes providing information to each group member about the status and activities of the other collaborators at a given time.

  • Workspace awareness. This emphasizes the fact that awareness generally emerges when people share a space. In other words, this kind of awareness refers to a common space that the group members share and where they can bring and discuss their findings and create a common product.

  • Contextual awareness. This type of awareness relates to the application domain, rather than the users. Here, we want to identify what content is useful for the group and what the goals are for the current project.

  • Peripheral awareness. This refers to the human ability to process information at the periphery of the attention, with a very low overhead. In other words, peripheral awareness relates to the information that should be kept separate (on their periphery) from what a participant is currently viewing or doing.

We started our investigation on the issue of awareness by posing the following questions.

  • How should different kinds of awareness be implemented (mappings from the kind of awareness to components)?

  • How should these awareness components be presented to the user (design)?

  • What are the costs and benefits associated with each of these awareness components (evaluation)?

As far as the design goes, several design decisions are already made for Coagmento based on previous works (e.g., Shah, Marchionini, & Kelly, 2009; Shah, 2010a), cognitive walkthroughs, pilot runs, and general understanding of creating a familiar, easy to use, and accessible interface. Therefore, the primary objective of the present work is not to study the optimal design or compare different designs for a CIS system, but to evaluate the impact of different awareness components for their effectiveness, efficiency, usefulness, and usability in supporting collaborative activities and inducing higher levels of satisfaction and engagement in collaboration.

Different CIS systems have different ways of providing awareness to the collaborators, depending on the domain and the kind of application. Take, for example Ariadne (Twidale, Nichols, & Paice, 1995), developed to support the collaborative learning of database browsing skills. To facilitate complex browsing processes in collaboration, Ariadne presents a visualization of the search process. This visualization comprises thumbnails of screens, looking like playing cards, which represented command-output pairs. Any such card can be expanded to reveal its details. The support for awareness, in this case, is driven by the specific domain (library) and application (catalogue search).

Search Together (Morris & Horvitz, 2007), on the other hand, was based on information seeking (application) on the Web (domain). SearchTogether instantiates awareness in several ways, one of which is per-user query histories, which is done by showing each group member's screen name, his or her photo, and queries in the “Query Awareness” region. The access to the query histories is immediate and interactive, as clicking on a query brings back the results of that query from when it was executed. The authors identified query awareness as a very important feature in collaborative searching, which allows group members to not only share their query terms but also learn better query formulation techniques from one another. Another component of SearchTogether that facilitates awareness is the display of page-specific metadata. This region includes several pieces of information about the displayed page, including group members who viewed the given page, and their comments and ratings. The authors claim that such visitation information can help one to choose to either avoid a page already visited by someone in the group, to reduce the duplication of efforts or, perhaps, visit such pages, as they provide a sign of promising leads as indicated by the presence of comments and ratings. Thus, SearchTogether provides awareness support via person, query, or Web page.

The design of our CIS system Coagmento (described below) was greatly influenced by SearchTogether, but unlike SearchTogether, Coagmento provides tools and support for not only searching and sharing but also organizing and synthesizing information. In addition to query and results awareness, Coagmento incorporates contextual and workspace awareness. This allowed us to explore the impact of various kinds of awareness in a CIS environment.

To our knowledge, there has not been any recent study of online collaborative information seeking that looks at awareness in the context of different interface designs. Our reported work here, thus, is highly valuable in adding to our understanding about awareness in CIS.

Coagmento—A Tool for Collaborative Information Seeking

We have developed Coagmento (Shah, 2010b), a Firefox plug-in that helps multiple people, working in collaboration, to communicate and to search, share, and organize information. Coagmento allows its users to be colocated or remote, working synchronously or asynchronously. The design of Coagmento is based on several personal interviews, mock-ups, cognitive walkthroughs, and pilot runs (Shah et al., 2009).

A screenshot of Coagmento is given in Figure 1. As we can see, it includes a toolbar and a sidebar. The toolbar has several buttons that help one collect information and be aware of the progress in a given collaboration. The toolbar has three major parts as follows:

thumbnail image

Figure 1. A screenshot of Coagmento with enhanced views of its toolbar and sidebar.

Download figure to PowerPoint

  • Buttons for collecting information and making annotations. These buttons help one save or remove a Web page, make annotations on a Web page, and highlight and collect text snippets.

  • Page-specific statistics. The middle portion of the toolbar shows various statistics, such as the number of views, annotations, and snippets, for the displayed page. A user can click on a given statistic and obtain more information. For instance, clicking on the number of snippets information will bring up a window that shows all the snippets collected by the collaborators from the displayed page.

  • Task-specific statistics. The last portion of the toolbar displays task name and various statistics, including number of pages visited and saved, about the current task. Clicking on that portion brings up the workspace, where one can view all the collected objects (pages and snippets) brought in by the collaborators for that task.

The sidebar features a chat window, under which there are three tabs with the history of search engine queries, saved pages, and snippets. With each of these objects, the user who created or collected that object is shown. Anyone in the group can access an object by clicking on it. For instance, one can click on a query issued by anyone in the group to rerun that query and bring up the results in the main browser window.

In the present design of Coagmento, different kinds of awareness discussed earlier are mapped to system components as follows.

  • Group awareness is facilitated by providing information about one's partner's current status (online/offline) and activities (e.g., document or rank list being viewed).

  • Workspace awareness is given by providing a common space, where the group members can see and compile their collective results (described later with Figure 5).

  • Contextual awareness is instantiated by display of information about the task and the goals.

  • Peripheral awareness is given by presenting a history of various actions, such as documents viewed and queries used as well as products such as saved documents and snippets, in a way that does not interrupt a user's current activity.

The study reported here varied conditions that included or excluded these awareness components in certain configurations. More specifically, the study manipulated peripheral awareness in three different ways by providing: (a) no peripheral awareness, (b) personal peripheral awareness, and (c) group peripheral awareness. The value of focusing on peripheral awareness is inspired by some of the works in the CSCW literature. For instance, Bly et al. (1993), with their experiments in media space, found that although it was seemingly invisible, peripheral awareness was the most useful one. Gaver (1992) also argued for the provision of peripheral awareness that provides unobtrusive information of the people, artifacts, and environment, while working on a collaborative project.

Method

  1. Top of page
  2. Abstract
  3. Introduction
  4. Background
  5. Method
  6. Analysis
  7. Design Implications
  8. Conclusion
  9. Acknowledgments
  10. References

To address the issue of awareness in CIS, we conducted a user study, in which we invited 42 pairs of participants to work on two different tasks over two separate sessions. The details of this study are given below.

Participants

We recruited 84 participants in 42 pairs from the University of North Carolina at Chapel Hill. These participants were asked to come to the lab for two different sessions, which were 1 to 2 weeks apart. The participants were able to choose the day and time convenient to them. Because the participants had to sign up in pairs, both the participants in a given pair already knew each other. In addition to this, we required that the participants in a given pair should have done some collaborative work with each other before, thus, making sure they not only know each other, but also are comfortable working with each other on a collaborative project. The approval of a pair's participation in this study was based on these requirements.

Of the 84 participants, 27 were male and 57 were female, and their ages ranged from 17 to 50 years with a median of 21 years. Several of the pairs were coworkers or spouses. A majority of the participants were undergraduate or graduate students, while a few were university employees. Participants were compensated $25 each for their participation in two sessions and competed for two Apple iPods presented to the team with the best performance (based on a scoring rubric that took into account the amount and kind of information collected, and the level of coordination the team had).

Conditions

The participants were assigned to one of the three conditions randomly. These conditions were defined based on the provision of different levels of awareness-related support to the participants, which are as follows:

  • 1.
    Baseline: support of contextual awareness (current task name and topic description) and workspace awareness (where the team can compile their final report; Figure 2).
  • 2.
    'Personal peripheral awareness: support of personal history (documents and snippets saved, queries used, etc.) in addition to the support provided in the baseline (Figure 3).
  • 3.
    Group peripheral awareness: support of group history (documents and snippets saved, queries used, etc.) in addition to the support provided in the baseline (Figure 4).
thumbnail image

Figure 2. Interface for condition 1, baseline.

Download figure to PowerPoint

thumbnail image

Figure 3. Interface for condition 2, personal peripheral awareness.

Download figure to PowerPoint

thumbnail image

Figure 4. Interface for condition 3, group peripheral awareness.

Download figure to PowerPoint

Thus, the main independent variable here was the kind of peripheral awareness provided (1=none, 2=personal, 3=group). All the conditions had communication, contextual awareness, and workspace awareness. No condition had the traditional group awareness (where a participant can see exactly what another collaborator is doing at a given time). This was because of the following two main reasons. For one, because the participants were required to work on the same task at the same time, they knew the other person was not only online, but also working on the same task. This realization eliminated the need to know the immediate status of one's collaborator. Furthermore, introducing this additional condition would weaken the statistical power of the study. Therefore, we decided to keep the traditional group awareness feature omitted from this study interface.

Note that peripheral awareness does not indicate the nature of the information, but how it is presented (at the periphery). For our purpose, we divided such awareness into personal and group peripheral awareness, corresponding to the information relating to personal and group history, respectively. Both conditions 2 and 3 had the same components and presentations for instantiating peripheral awareness, but the information displayed in each of them was different: condition 2 interface included a user's personal information only, whereas condition 3 interface provided the team's cumulative information.

To support workspace awareness, each participant was given access to a common workspace (Figure 5). This workspace is updated in real time and shows the collected objects (Web pages and snippets).

Our random assignment of each pair of participants yielded 14 pairs for each of the three conditions.

thumbnail image

Figure 5. A screenshot of Coagmento workspace.

Download figure to PowerPoint

Sessions

We brought each individual pair of participants to the lab for two sessions that were 1 to 2 weeks apart. Each session lasted about one and a half hours. The flow for each session is depicted in Figure 6.

thumbnail image

Figure 6. Flow of the two sessions.

Download figure to PowerPoint

During the first session, the participants were shown a video tutorial that demonstrated the use of Coagmento and the process of collecting relevant information (snippets of text). After the tutorial, the participants were placed in different rooms so that they could not talk to each other directly or see what the other person was doing (Figure 7). Both participants used typical mid-end PC workstations, running Windows XP, with Ethernet connectivity and 19-inch monitors. A researcher was stationed so that he could see both participants. Once the participants logged in, they filled out a demographic questionnaire and began working on the first task.

thumbnail image

Figure 7. Subjects in a study session.

Download figure to PowerPoint

As discussed below, the tasks were simulated work tasks (Borlund, 2003). About 20 minutes into their work, the researcher sent out a message via the sidebar chat asking them to stop the task and fill in an online questionnaire, as if their supervisor had requested an update. Once both the participants finished their individual questionnaires, they were asked to start working on the second task. The participants were once again interrupted about 20 minutes later, and asked to complete the same questionnaire.

After the post-task questionnaire for the second task, the participants were asked to fill in an end-of-session questionnaire. After the participants submitted their individual end of session questionnaires, they were brought together and asked a few questions.

For the second session, the participants were given a refresher of the system and shown how to compile their final report by grouping their collected snippets into different categories for each of the two tasks. The categories were presented in the task statement and corresponded to different aspects of the work task (see task statements below). The participants were then asked to take their places in the room other than the one they had the last time, to take care of any bias the participants may have for the machine or the room they used.

After 20 minutes of additional work on task one, they completed the post-task questionnaire and were asked to organize their collected snippets by placing each relevant snippet into one of the categories for a given task. When they had finished organizing their snippets, they worked through the second task, including collecting their information, post-task questionnaire, and organizing the snippets. Upon completing both the tasks, the participants completed an exit questionnaire, after which they were brought together and interviewed.

Tasks

The participants were asked to collect relevant information for two exploratory tasks, which were designed to be realistic work tasks, that might be of relevance and interest to the participant pool (Borlund & Ingwersen, 1999). Rather than asking participants to create their own organizations for the pertinent snippets, the task statements identified specific issues that should be addressed, and these issues were used as organizing bins for the collected snippets.

The task descriptions as given to the subjects are provided in the following sections.

Task 1: Economic Recession

A leading newspaper has hired your team to create a comprehensive report on the causes and consequences of the current economic recession in the U.S. As a part of your contract, you are required to collect all the relevant information from any available online sources that you can find.

To prepare this report, search and visit any Web site that you want and look for specific aspects as given in the guideline below. As you find useful information, highlight and save relevant snippets. Later, you can use these snippets to compile your report. You may also want to save the relevant Web sites as bookmarks, but remember, your main objective here is to collect as many relevant snippets as possible.

Your report on this topic should address the following issues: reasons behind this recession, effects on some major areas, such as healthcare, home ownership, and financial sector (stock market), unemployment statistics over a period of time, proposal, execution, and effects of the economy stimulation plan, and people's opinions and reactions on economy's downfall.

Task 2: Social Networking

The College Network News Channel wants to do a documentary on the effects of social networking services and software. Your team is responsible for collecting various relevant information (including statistics) from the Web. As a part of your assignment, you are required to collect all the relevant information from any available online sources that you can find.

To prepare this report, search and visit any Web site that you want and look for specific aspects as given in the guideline below. As you find useful information, highlight and save relevant snippets. Later, you can use these snippets to compile your report. You may also want to save the relevant Web sites as bookmarks, but remember, your main objective here is to collect as many relevant snippets as possible.

Your report on this topic should address the following issues: emergence and spread of social networking sites, such as MySpace, Facebook, Twitter, and del.icio.us, statistics about popularity of such sites (How many users? How much time they spend? How much content?), impacts on students and professionals, commerce around these sites (How do they make money? How do users use them to make money?), and examples of usage of such services in various domains, such as healthcare and politics.

Analysis

  1. Top of page
  2. Abstract
  3. Introduction
  4. Background
  5. Method
  6. Analysis
  7. Design Implications
  8. Conclusion
  9. Acknowledgments
  10. References

The core question of the study described here is to look at the impact of awareness in collaborative information-seeking projects. The three conditions we used were defined based on the amount and kind of awareness provided to the collaborators. To measure how aware the participants were at different points, we employed several instruments. In this section, we present analysis based on studying the question of awareness using these instruments. In addition to measuring awareness directly, we also used several other outcome measures, such as productivity, user satisfaction, and engagement, for a more comprehensive understanding of the impacts of awareness support in CIS.

For the purpose of the discussion and the analysis presented here, we have collapsed both the tasks and the sessions together. Our initial analysis of these two independent variables found no statistically significant differences between the tasks or sessions for the perceived and reported awareness measures, and so they are combined in what follows. When analyzing a questionnaire, if creating an index that combines all of the questions on that questionnaire was appropriate, then that index's statistical reliability was checked, and if reliability statistic was α,>,0.7, then such an index was created by averaging the responses for all the questions. One-way analysis of variance (ANOVA) on this index is done, and if found significant with p<0.05, then Sheffe's post hoc test was executed. If creating an index was not appropriate or its statistical reliability was not sufficient, or if an item for analysis was log data, then straight one-way ANOVA was done, followed by Sheffe's post hoc test, if appropriate. Note that each condition had 14 groups, with 28 individuals. Therefore, there was a total of 42 groups, with 84 individuals.

Productivity

To begin our analysis, we first looked at the productivity of each team. This was measured in two ways: looking at each participant's activities (Table 1) and finding the Web pages overlapped between the collaborators in a given pair (Table 2).

Table 1. Average use/collection of various objects by each user for each condition.
 Condition 1Condition 2Condition 3
 MeanSDMeanSDMeanSD
  • Note. SD=standard deviation.

  • *

    *Difference across conditions was statistically significant (p<0.05).

  • For one-way analysis of variance, between groups degree of freedom (df)=2 and within groups df=81.

Total queries (F=2.391, p<0.098)44.50203749.6118.4658.0729.92
Unique queries* (F=3.177, p<0.149)24.1110.3729.119.9431.9314.44
Total webpages viewed (F=1.952, p<0.149)124.7131.22142.8645.10144.6447.15
Unique webpages viewed (F=2.788, p<0.067)79.4316.9592.8927.0092.6127.68
Webpages saved (F=0.489, p<0.615)13.545.7514.868.0415.548.95
Snippets collected (F=0.141, p<0.0868)19.9514.7318.7913.0719.809.38
Table 2. Summary statistics of Web pages overlapped between the collaborators in a given pair.
 Condition 1Condition 2Condition 3
 MeanSDMeanSDMeanSD
  1. Note. SD=standard deviation.

  2. For one-way analysis of variance, between groups degree of freedom (df)=2 and within groups df=39.

Webpages overlapped within team (F=0.036, p<0.964)4.003.964.292.274.072.16

We found that participants in condition 3 used significantly more unique queries than those in condition 1 (p<0.05). Given that there was no difference between these two conditions for the total number of queries used, we can say that the participants in condition 3 had much less overlap in their query usage and could try more unique searches for the task at hand.

Looking at the overlap of Web page visits between the two participants in a team (Table 2), we see that those in condition 2 had slightly more overlap than the other two conditions. However, these differences were not found to be statistically significant.

These observations inform us that although it appears that the teams in each condition are doing the same amount of work, condition 3 teams are clearly able to explore more venues of information overall. This can be attributed to the kind of awareness provided. For example, a participant in condition 1 has no support for knowing the search queries he or his teammate used in the past. Therefore, he is likely to repeat a search even when it is not needed. A participant in condition 2, on the other hand, can avoid repeating his own searches, unless necessary because of the personal peripheral awareness support. A participant in condition 3 will have knowledge about his as well as his teammate's searches. This explains why greater differences were found between conditions 1 and 3 for unique query usage, and not among other pairs of conditions.

Another important aspect of the assigned tasks was the snippets. As shown in Table 1, no significant difference was found in the number of snippets collected by the teams in different conditions. Much of the snippets collected can be attributed to personal motivation. The team that collected the most number of snippets was in condition 1. Given that time was limited, the tasks exploratory, and a prize to be won for those collecting many snippets, this team quickly divided the tasks and went through as many Web pages as possible to collect numerous snippets. This team gathered 223 snippets combined for both the tasks during the two sessions. This was significantly higher than the average, which was about 78. The minimum number of snippets collected was 38, by a team in condition 1.

To summarize, it is hard to claim difference in productivity among different conditions. On the other hand, through query usage, it was found that teams in condition 3 were being more effective in their work, that is, they managed to explore more volume of information with the same amount of time and work.

Perceived Awareness

To obtain feedback on how aware the participants felt, the following statements, inspired by Govern and Marsch (2001), were presented individually to the participants at the end of each task for both sessions. The participants were asked to select a value between 1 (strongly disagree) to 7 (strongly agree) for these statements.

  • equation image

Because creating an index is appropriate for this questionnaire, and it was found to be statistically reliable with α=0.781, an index was created by averaging the responses on all seven questions for every individual. Table 3 reports various statistics for this index across all the conditions. No significant difference was found among the three conditions. This indicates that as far as being conscious of the situation around them was concerned, participants in each condition were similarly aware.

Table 3. Index for perceived situational awareness questions.
 Condition 1Condition 2Condition 3
 MeanSDMeanSDMeanSD
  1. Note. SD=standard deviation.

  2. For one-way analysis of variance on each question, between groups degree of freedom (df)=2 and within groups df=333.

Index for perceived situational awareness (F=1.653, p<0.193)4.690.8594.561.0624.790.924

In addition to these post-task questions, the participants were also asked to rate two specific questions, given below, relating to personal and group awareness at the end of each session.

  • equation image

The average responses on the scale, 1 (strongly disagree) to 7 (strongly agree), are shown in Table 4 for each condition. Condition 2 participants reported higher personal awareness (Q1), even higher than those in condition 3 (p<0.018). This is not surprising given that condition 2 had exclusive support for personal history. But when it comes to being aware of the team, condition 3 participants reported significantly higher values than those in condition 1 (p<0.001) or 2 (p<0.001). This is, once again, an indication that the participants in condition 3 were much more aware of the status of the team and the task throughout their collaborative endeavor.

Table 4. Summary statistics for responses on perceived awareness questions at the end of each session.
 Condition 1Condition 2Condition 3
 MeanSDMeanSDMeanSD
  • Note. SD=standard deviation.

  • *

    *Difference across conditions was statistically significant (p<0.05).

  • For one-way analysis of variance on the index, degree of freedom (df)=2 and within groups df=165.

Q1. personal awareness* (F=4.426, p<0.013)5.730.8426.120.9555.591.141
Q2. group awareness* (F=19.221, p<0.001)4.551.2053.951.5315.501.250

The questions listed above asked the participants to report their level of awareness about certain aspects directly. This may not be enough to tell the whole story about awareness. We, therefore, asked the participants more specific questions about the status of their ongoing task, as described next.

Task Status Awareness

As listed in the method section, the participants were interrupted after about 20 minutes for a given task and asked to report the task status. This status was obtained by asking the following questions.

  • equation image

We will first see how much the collaborators of a given team were in agreement of reporting the status of the task by analyzing questions 2, 4, 6, and 8 in the above-mentioned questionnaire. It was found that creating an index was appropriate and statistically reliable (α=0.759). Table 5 reports the statistics with this index for average differences (absolute values) in the responses to these questions for a given team. As we can see, the teams in condition 3 have the least difference for these questions, and these differences were significant compared with condition 1 (p<0.024) and condition 2 (p<0.001). In other words, these results indicate that the participants in condition 3 were highly coordinated when it came to being aware of the group's history and progress. It is also interesting to note is that even the teams in condition 1 had smaller differences than those in condition 2 (p<0.046).

Table 5. Index for absolute difference between two collaborators' reported numbers for group's status.
 Condition 1Condition 2Condition 3
 MeanSDMeanSDMeanSD
  • Note. SD=standard deviation.

  • *

    *Difference across conditions was statistically significant (p<0.05).

  • For one-way analysis of variance on the index, degree of freedom (df)=2 and within groups df=165.

Index for group reporting difference* (F=13.827, p<0.001)2.383.2473.994.8960.5980.756

Because the participants were allowed to chat with each other while filling in these values, we cannot simply look at their differences and comment about the strength of their collaboration. We also need to consider the cost associated with achieving the coordination. This was done by analyzing the time and the number of messages that they exchanged while completing this questionnaire.

The results are reported in Table 6. It turns out that there was no statistically reliable difference between conditions 1 and 2 for the amount of time used; however, participants in condition 1 exchanged significantly more messages than those in condition 2 (p<0.018). It was also found that condition 3 had significantly less overhead in terms of communication compared with conditions 1 (p<0.001) and 2 (p<0.001), while filling in this status report. They also spent less time compared with condition 1(p<0.001) and 2 (p<0.001). This shows that the participants in condition 3 were well aware of the status of their tasks and could report it without spending much time or communication.

Table 6. Average amount of time spent and messages exchanged by each participant while reporting project status.
 Condition 1Condition 2Condition 3
 MeanSDMeanSDMeanSD
  • Note. SD=standard deviation.

  • *

    *Difference across conditions was statistically significant (p<0.05).

  • For one-way analysis of variance on the index, degree of freedom (df)=2 and within groups df=333.

Avg. time (sec)* (F=1.714, p<0.001)198.18114.149180.12130.626123.0471.000
Avg. no. of messages* (F=39.981, p<0.001)13.9411.96310.1210.5762.376.103

Perceived Awareness Versus Reality

Next, let us look at how close participants' reporting about task awareness was to the reality. For this purpose, eight questions (given in the previous subsection) on which a participant was asked to report various numbers regarding personal or team history on the task were considered. The actual numbers were then calculated using the log data and compared with the reported numbers. The results of this comparison are presented in Tables 7 and 8.

Table 7. Average difference between reported numbers and real numbers for questions on personal progress.
 Condition 1Condition 2Condition 3
 MeanSDMeanSDMeanSD
  • Note. SD=standard deviation.

  • *

    *Difference across conditions was statistically significant (p<0.05).

  • For one-way analysis of variance on each response, degree of freedom (df)=2 and within groups df=333.

Webpages viewed* (F=161.194, p<0.001)16.6810.8550.351.4865.654.960
Webpages saved* (F=11.331, p<0.001)2.442.6601.121.6931.611.812
Snippets saved* (F=11.832, p<0.001)2.864.7210.622.0232.163.349
Queries used* (F=23.192, p<0.001)2.292.3990.461.3141.832.380
Table 8. Average difference between reported numbers and real numbers for questions on group progress.
 Condition 1Condition 2Condition 3
 MeanSDMeanSDMeanSD
  • Note. SD=standard deviation.

  • *

    *Difference across conditions was statistically significant (p<0.05).

  • For one-way analysis of variance on the index, degree of freedom (df)=2 and within groups df=333.

Webpages viewed* (F=198.123, p<0.001)31.7518.3196.259.7281.564.152
Webpages saved* (F=49.665, p<0.001)4.273.4353.663.3350.721.209
Snippets saved* (F=29.554, p<0.001)5.957.7892.724.7520.491.599
Queries used* (F=30.042, p<0.001)4.324.2152.093.6390.911.585

The following observations can be made from these results:

  • The participants in condition 2 did significantly better than condition 1 while reporting personal numbers on Web pages viewed (p<0.001), Web pages saved (p<0.001), snippets collected (p<0.001), and queries used (p<0.018). In fact, condition 2 also did better than condition 3 on Web pages viewed (p<0.001), snippets collected (p<0.005), and queries used (p<0.001). This makes sense, as they were the only ones with exclusive support for personal awareness.

  • However, the participants in conditions 1 and 2 did not have group awareness support. Therefore, the participants in condition 3 could report the numbers for group progress that were significantly closer to the reality compared with those reported by condition 1 or 2. Condition 3 outperformed condition 2 on Web pages viewed (p<0.017) and saved (p<0.001), snippets collected (p<0.008), and queries used (p<0.030). They also did better than those in condition 1 on all of these fronts (p<0.001).

  • The participants in condition 3 did significantly better, even on personal awareness-related questions on Web pages viewed (p<0.001) and saved (p<0.013), compared with those in condition 1. The participants in condition 3 did not have explicit support for personal awareness, and they were in a better position to make educated guesses (and sometimes actual counts) using the group awareness information, than those in condition 1.

It is clear from these observations that the condition 2 interface provides the most suitable support for personal awareness, whereas condition 3 has the best support for group awareness. An ideal interface may include group information with the ability to switch to personal information as needed. In other words, such an interface would provide more control for the awareness to the user. During the interviews at the end of each session, a majority of participants in condition 3 expressed their desire to have this ability of switching to personal awareness mode, while retaining the group awareness support.

Workspace Awareness

In each of the conditions, the participants were given the same kind of workspace (Figure 5). This workspace included the pages and the snippets saved by the team (both the participants). Thus, even the participants in condition 1, who had no personal or group awareness, could access this workspace and see where the team stood at a given point. Given that the workspace was a common component among all the conditions, we were curious to see if having peripheral (personal or group) awareness would change how often one accesses this workspace.

We found that on average (Table 9), the participants in condition 1 accessed the workspace more than those in conditions 2 (p<0.023) and 3 (p<0.001). There was no statistically significant difference for workspace accesses between conditions 2 and 3. This indicates that the participants without any kind of awareness support had to access the common space more often to stay updated about personal and team progress, thus, further providing support for the usefulness of the peripheral awareness in collaborative projects.

Table 9. Average number of times workspace accessed by each participant.
 Condition 1Condition 2Condition 3
 MeanSDMeanSDMeanSD
  • Note. SD=standard deviation.

  • *

    *Difference across conditions was statistically significant (p<0.05).

  • For one-way analysis of variance on the index, degree of freedom (df)=2 and within groups df=81.

Avg. no. of times workspace accessed* (F=12.818, p<0.001)11.074.8837.934.0275.433.511

Physical Effort

To measure the amount of physical work one did during this study, the logs were mined, and the key actions that every team performed were enumerated. These key actions are as follows: viewing or saving a Web page, executing a query, collecting a snippet, making or viewing an annotation, clicking on any item in the sidebar, and accessing the workspace. A summary of the number of such actions for each condition is reported in the second column of Table 10. As we can see, the participants in condition 3 performed more actions than those in condition 1 (p<0.046).

Table 10. Average number of key actions, chat actions, and combined actions taken by each team.
 Condition 1Condition 2Condition 3
 MeanSDMeanSDMeanSD
  • Note. SD=standard deviation.

  • *

    *Difference across conditions was statistically significant (p<0.05).

  • For one-way analysis of variance on the index, degree of freedom (df)=2 and within groups df=39.

Key actions* (F=3.393, p<0.044)460.7978.861529.5095.191571.14152.191
Chat actions* (F=7.592, p<0.002)219.5081.408166.3679.369103.7175.209
Combined actions (F=0.097, p<0.908)680.29115.133695.8692.494674.86172.400

One important action that is omitted from these key actions is chat. Let us first look at chat action itself. The third row in Table 10 shows the average number of chat messages sent between the teammates for each condition. As we see, now the relationships between different conditions are reversed. The participants in condition 1 exchanged significantly more messages than those in condition 3 (p<0.002). It was already shown (Table 6) that the participants in conditions 1 and 2 exchanged significantly more chat messages while reporting the team's progress. The observations reported in Table 10 complements it by indicating a significantly higher volume of communication for these two conditions throughout the sessions. The implications of exchanging a higher number of messages are difficult to state; but, later in this section, we will see if the greater amount of communication resulted in a higher level of satisfaction or engagement among the participants.

Now, if we include chat into the other key actions (row four), then we find that different conditions are not very far from each other in terms of the number of actions performed.

Synthesizing the observations from these three columns, we can say that while the teams in different conditions took similar number of key actions, those in condition 3 explored more information (and probably reflected more on that information as well) and spent less effort in communicating with each other. Of course, not all communication can be considered as cost; but, as we saw earlier, the teams in conditions 1 and 2 sent significantly more messages (and spent more time) than those in condition 3, who tried to coordinate with and update each other.

Cognitive Effort

Although it seems that providing more (and appropriate) support for awareness allows the users to be more aware of their collaborators and progress, we also need to consider the effect of such support on other factors of usability. One such factor is perceived effort, primarily mental effort or cognitive load.

We used six questions based on NASA's tax load index for measuring the perceived cognitive load.33 The questions are given below. These questions were answered at the end of each task in each session.

  • equation image

The participants were asked to rate each of the six questions, from 1 (very low) to 7 (very high). An index was created for these questions (α=0.730) and is reported in Table 11.

Table 11. Index for cognitive load questionnaire filled by individuals.
 Condition 1Condition 2Condition 3
 MeanSDMeanSDMeanSD
  1. Note. SD=standard deviation.

  2. For one-way analysis of variance on the index, degree of freedom (df)=2 and within groups df=333.

Index for cognitive load questions (F=2.178, p<0.115)3.210.8253.190.8023.411.005

No statistically significant differences were found among the three conditions. This informs us that even those with a more complex interface, such as condition 3, felt no more mentally loaded than those with a simple interface, such as condition 1. This is good news for system designers, as it indicates that we could provide additional support for CIS without adding any burden to the user.

Given that the participants in each condition did a similar amount of work, the question is: Were they happy doing it? To learn this, we employed two more questionnaires at the end of each session: one for engagement and another for ease of use and satisfaction.

Engagement

Because this study was done in an interactive environment, one important factor to analyze was how engaged the participants felt while working on their tasks. To obtain this feedback, the participants were presented with the following questionnaire at the end of each session. This questionnaire was adopted from the full instrument presented in (Ghani, Supnick, & Rooney, 2007).

  • equation image

An index was created for these eight questions (α=0.889) and reported in Table 12. It was found that the participants in condition 3 gave significantly higher ratings compared with those in condition 2 (p<0.015). This indicates that although the condition 3 interface was more complex, the participants felt more engaged while working in collaboration with the system.

Table 12. Index for engagement questions responded by individuals.
 Condition 1Condition 2Condition 3
 MeanSDMeanSDMeanSD
  • Note. SD=standard deviation.

  • *

    *Difference across conditions was statistically significant (p<0.05).

  • For one-way analysis of variance on the index, degree of freedom (df)=2 and within groups df=165.

Index for engagement* (F=4.344, p<0.014)5.580.7195.350.8865.810.825

These results indicate that engagement in a user-centric CIS environment depends on not only the method of providing awareness (here, peripheral awareness), but also the nature of that awareness (personal vs. group). Condition 3 mirrored condition 2 in terms of the design for providing awareness (at the periphery), but the kind of awareness provided in the same screen space was different (group, instead of personal). This allowed the participants in condition 3 to carry on their tasks without being distracted and, at the same time, to be aware of the status of their teammates and the overall task.

Ease of Use and Satisfaction

Finally, to obtain feedback on ease of use and satisfaction, we presented the participants with the following statements at the end of each session and asked them to rate them on a scale of 1 (strongly disagree) to 7 (strongly agree). The questionnaire was derived from the original Computer System Usability Questionnaire (Lewis, 1995),44 removing those questions that were not relevant for this study. Responses to these questions shed light on their perceived ease of use and satisfaction.

  • equation image

An index was created for the 13 questions (α=0.986), which is reported in Table 13. We found it interesting to see that compared with condition 2, participants in condition 1 reported higher values (p<0.010). One of the implications of these results is that although the participants in both the conditions liked the interface provided to them, they had differences in the expected functionalities and satisfaction from the system. Because they had personal awareness support, they realized how valuable it could be to have similar awareness about their teammates.

Table 13. Index for ease of use and satisfaction questions responded by individuals.
 Condition 1Condition 2Condition 3
 MeanSDMeanSDMeanSD
  • Note. SD=standard deviation.

  • *

    *Difference across conditions was statistically significant (p<0.05).

  • For one-way analysis of variance on the index, degree of freedom (df)=2 and within groups df=165.

Index for ease of use and satisfaction* (F=5.065, p<0.007)5.990.6615.041.7895.312.120

No difference was found between conditions 1 and 3, and 2 and 3. This, once again, indicates that a complex interface, such as condition 3, can do equally well as a simple interface, such as condition 1, when it comes to providing ease of use and satisfaction. Even though condition 3 had more information on the interface, the organization of that information was similar to condition 2 (on the periphery). This allowed the participants in condition 3 to carry on their tasks without being distracted and, at the same time, to be aware of the status of their teammates and the overall task. In other words, the condition 3 interface could provide a better support for awareness without sacrificing ease of use and satisfaction.

Design Implications

  1. Top of page
  2. Abstract
  3. Introduction
  4. Background
  5. Method
  6. Analysis
  7. Design Implications
  8. Conclusion
  9. Acknowledgments
  10. References

There are larger implications for designing a CIS system based on the kind of awareness desired. To understand this, we have placed a number of systems and tools that support CIS in Figure 8. This is based on a classical way of organizing collaborative or groupware systems (Rodden, 1991).

thumbnail image

Figure 8. CIS systems organized according to time and space aspects.

Download figure to PowerPoint

The placement of a CIS system on this figure has implications for its implementation, functionalities, and evaluation. For instance, Adobe Connect facilitates online meetings, where the participants can share and discuss information. Such an environment will fall under Synchronous-remote collaboration in Figure 8. Thus, this environment needs to have a way to connect remote participants in real time and a shared space for exchanging and processing information. Our proposed system, Coagmento, is designed to provide support for collaborators who work in synchronous or asynchronous mode and are primarily remotely located. This study simulated the situation of remotely located synchronous collaboration. We asked the participants during the interviews what they would like to see more of in this system if they were using it for their real-life projects, and they offered many interesting suggestions. Because they may not be working synchronously with their teammates most of the time, they desired to have a shared notepad, such as ShrEdit (Olson, Storrøsten, & Carter, 1993). During the study, the participants were aware of their teammate being online and ready to respond. However, when there is no prerequisite of working in synchronous mode in collaboration, the participants asked for a way to know when their teammates were online. They also wanted more active notifications of chat messages and other critical actions, such as collecting relevant information. As we can see, although some of these components that provide awareness were not required for our lab study (colocated, synchronous), they were desired in other situations.

We started our study with a hypothesis that more awareness would help in making the collaborative experience more productive, engaged, and satisfactory for users. However, we found that although the condition with the most awareness performed best overall, the condition that fell in between the highest and the no awareness systems did the worst. This indicates the importance of providing an interface that matches the complexity and nature of the task. Because collaboratively seeking information in an interactive and multisession setting is a relatively more complex activity than an individual seeking information in a single session, condition 3 provided appropriate interface for the situation. Condition 1 incurred significantly more cost for coordinating events and kept the participants less engaged and aware, but the participants reported high level of satisfaction and ease of use. Condition 2, on the other hand, had complex interface and did not provide enough added benefits as condition 3 did. Thus, it achieved low ratings for usability, while not offering any significant benefits over the baseline.

Overall, the results of this study suggest that designers provide support for group awareness. Although this study focused on synchronous collaboration, it is likely that group awareness support will be even more important when people search together asynchronously. The results suggest that the added complexity of group awareness can be implemented in such a way to avoid adding cognitive load to the users.

Conclusion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Background
  5. Method
  6. Analysis
  7. Design Implications
  8. Conclusion
  9. Acknowledgments
  10. References

Web-based search services have become ubiquitous in daily life and are beginning to move beyond the simple query-based lookup and result list paradigm. Collaborative information seeking has emerged as one important kind of extension to this paradigm. In this article, we examined awareness in CIS projects, where the collaborators were working remotely in synchronous mode. We described a CIS system, Coagmento, which was inspired by previous work on user-centric, collaborative interfaces. Using three versions of this system, each of which provided different kinds and levels of awareness, we conducted a user study to address various questions related to role and impact of awareness in CIS. This study generated a large amount of data across 84 participants, who conducted two exploratory search tasks spanning two sessions, and focused attention on how awareness functionalities affect collaborative search. The data strongly demonstrate that basic group awareness components offer substantial advantages to collaborative search without adding new burdens to users. In particular, we found that although having personal awareness support could be useful in multisession, exploratory search tasks, it is not enough for collaborative projects. In our study, the participants without any kind of awareness support reported comparable loads and satisfactions to those with full team awareness support; however, these searchers executed far fewer queries, examined fewer pages, and used much more chat to complete their tasks. They were able to work without the group awareness support by using their memories and instant communication capabilities. It is likely that people could find having team awareness even more valuable and effective in many practical situations where they are not working synchronously.

Another way to summarize the results of the reported study is that we need to provide adequate and appropriate interface depending on the task. The version of Coagmento used in condition 3 had complex interface with proper awareness support, but because the CIS tasks and setup were complex enough, the participants could benefit from the added functionality provided by this interface. The same interface may not be suitable for a single-session, information-seeking task by an individual. The condition 2 interface had the similarly complex interface, but not enough awareness information. This resulted in lower levels of engagement and awareness among the participants. These results can be summarized as follows: (a) personal awareness is required but not sufficient; (b) group awareness support helps the participants be more efficient, engaged, and aware without raising cognitive load; and (c) one needs to provide adequate and appropriate support for awareness given the task and the situation.

The work reported here demonstrated the need for good awareness support functions using Coagmento and should be helpful to designers of other CIS systems. The data collected here bear additional analysis, especially with respect to the dynamics within pairs of collaborators using the different awareness support services. The present work can be extended in several ways. One of them is by a field study that allows the participants to work without the limitations the controlled lab study had. The participants in a field study could work with the system as they please, creating their own projects of interest, and initiating collaborators with their colleagues and friends as they see fit. Running a field study over a long period of time (at least a few weeks) will also allow one to study long-term adoption effects, appropriation factor, and specialization with various features of the system. Specifically, regarding awareness support, group awareness can be highly desired in many real life situations, allowing the collaborators to be aware of each others' current status (online/offline, present working subtask, etc.).

Acknowledgments

  1. Top of page
  2. Abstract
  3. Introduction
  4. Background
  5. Method
  6. Analysis
  7. Design Implications
  8. Conclusion
  9. Acknowledgments
  10. References

This work was supported by the National Science Foundation under grant no. IIS 0812363. We thank all of our participants for their valuable time and inputs.

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. Background
  5. Method
  6. Analysis
  7. Design Implications
  8. Conclusion
  9. Acknowledgments
  10. References
  • Bly, S.A., Harrison, S.R., & Irwin, S. (1993). Media spaces: Bringing people together in a video, audio, and computing environment. Communications of the ACM, 36(1), 2846.
  • Borlund, P. (2003). The IIR evaluation model: A framework for evaluation of interactive information retrieval systems. Information Research, 8(3).
  • Borlund, P., & Ingwersen, P. (1999). The application of work tasks in connection with the evaluation of interactive information retrieval systems: Empirical results. In M.Dunlop, S.Draper, K.van Rijsbergen, I.Ruthven, (Eds.), Mira '99 (pp. 2946). British Computer Society Electronic Workshops in Computing.
  • Chalmers, M. (2002). Awareness, representation and interpretation. Computer Supported Cooperative Work, 11(3–4), 389409.
  • Dourish, P., & Bellotti, V. (1992). Awareness and coordination in shared workspaces. In Proceedings of the Conference on Computer-Supported Cooperative Work (CSCW) (pp. 107114). New York: ACM Press.
  • Dourish, P., & Bly, S.A. (1992). Portholes: Supporting awareness in a distributed work group. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems Conference on Computer-Supported Cooperative Work (CSCW) (pp. 541547). New York: ACM Press.
  • Farooq, U., Ganoe, C.H., Carroll, J.M., & Giles, C.L. (2009). Designing for e-science: Requirements gathering for collaboration in CiteSeer. International Journal of Human Computer Studies, 67, 297312.
  • Gaver, W.W. (1991). Sound support for collaboration. In Bannon, L.J., Robinson, M., & K.Schmidt (Eds.), Proceedings of European Conference on Computer Supported Cooperative Work (CSCW) (pp. 293308). Norwell, MA: Kluwer Academic Publishers.
  • Gaver, W.W. (1992). The affordances of media spaces for collaboration. In J.Turner, & R.Kraut (Eds.), Proceedings of the Conference on Computer-Supported Cooperative Work (CSCW) (pp. 1724). New York: ACM Press.
  • Gaver, W.W., Moran, T.P., MacLean, A., Lövstrand, L., Dourish, P., Carter, K., et al. (1992). Realizing a video environment: EuroPARC's RAVE system. In Bauersfeld, P., Bennett, J., & Lynch, G. (Eds.). Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems Conference on Human Factors in Computing Systems Conference on Human Factors in Computing Systems (pp. 2735). New York: ACM Press.
  • Ghani, J.A., Supnick, R., & Rooney, P. (1991). The experience of flow in computer-mediated and in face-to-face groups. Proceedings of International Conference on Information Systems, Minneapolis (pp. 229237). University of Minnesota.
  • Grudin, J. (1994). Groupware and social dynamics: Eight challenges for developers. Communications of the ACM, 37(1), 92105.
  • Gutwin, C., & Greenberg, S. (2002). A descriptive framework of workspace awareness for real-time groupware. Computer Supported Cooperative Work, 11(3–4), 411446.
  • Hansen, P., & Järvelin, K. (2005). Collaborative information retrieval in an information-intensive domain. Information Processing & Management, 41, 11011119.
  • Harper, R.H., Hughes, J.A., & Shapiro, D.Z. (1989a). The functionality of flight strips in ATC work. The report for the Civil Aviation Authority. Technical report, Lancaster Sociotechnics Group, Department of Sociology, Lancaster University.
  • Harper, R.H., Hughes, J.A., & Shapiro, D.Z. (1989b). Working in harmony: An examination of computer technology in air traffic control. In Proceedings of European Conference on Computer Supported Cooperative Work (CSCW) (pp. 7286). London: Gatwick.
  • Heath, C.C., & Luff, P. (1991). Collaborative activity and technological design: Task coordination in London underground control rooms. In Bannon, L.J., Robinson, M., & Schmidt, K., (Eds.), Proceedings of European Conference on Computer Supported Cooperative Work (ECSCW) (pp. 6580). Dordrecht, The Netherlandst: Kluwer Academic Publishers.
  • Heath, C., Svensson, M.S., Hindmarsh, J., Luff, P., & vom Lehn, D. (2002). Configuring awareness. Computer Supported Cooperative Work, 11(3–4), 317347.
  • Lewis, J.R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7(1), 5758.
  • Liechti, O., & Sumi, Y. (2002). Editorial: Awareness and the WWW. International Journal of Human Computer Studies, 56(1), 15.
  • Mantei, M.M., Baecker, R.M., Sellen, A.J., Buxton, W.A.S., & Milligan, T. (1991). Experiences in the use of a media space. In S.Robertson, G.Olson, & J.Olson, (Eds.), Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems (pp. 203208). New York: ACM Press.
  • Morris, M.R. (2007). Interfaces for collaborative exploratory Web search: Motivations and directions for multi-user design. Proceedings of ACM SIGCHI 2007 Workshop on Exploratory Search and HCI: Designing and Evaluating Interfaces to Support Exploratory Search Interaction (pp. 912). New York: ACM Press.
  • Morris, M.R., & Horvitz, E. (2007). SearchTogether: An interface for collaborative Web search. In Proceedings of the 20th annual ACM Symposium on User Interface Software and Technology (UIST '07) (pp. 310). New York: ACM Press.
  • Olson, J.S., Olson, G.M., Storrøsten, M., & Carter, M. (1993). Groupwork close up: A comparison of the group design process with and without a simple group editor. ACM Transactions on Information Systems, 11(4), 321348.
  • Pickens, J., Golovchinsky, G., Shah, C., Qvarfordt, P., & Back, M. (2008). Algorithmic mediation for collaborative exploratory search. In Proceedings of the 31st Annual ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 315322). New York: ACM Press.
  • Rodden, T. (1991). A survey of CSCW systems. Interacting with Computers, 3(3), 319353.
  • Schmidt, K. (2002). The problem with ‘awareness’: Introductory remarks on ‘awareness in CSCW.’ Computer Supported Cooperative Work, 11(3), 285298.
  • Shah, C. (2010a, February). Working in collaboration—What, why, and how? Paper presented at the Conference on Computer-Supported Cooperative Work (CSCW), 2010 Workshop on Collaborative Information Seeking, Savannah, GA. Retrieved June 14, 2010, from http://workshops.fxpal.com/cscw2010cis/submissions/tmp11.pdf
  • Shah, C. (2010b, February). Coagmento—A collaborative information seeking, synthesis and sense-making framework. Integrated demo at the Conference on Computer-Supported Cooperative Work (CSCW), 2010 Workshop on Collaborative Information Seeking, Savannah, GA.
  • Shah, C., Marchionini, G., & Kelly, D. (2009). Learning design principles for a collaborative information seeking system. In Extended Proceedings of ACM SIGCHI Conference (pp. 34193424). New York: ACM Press.
  • Simone, C., & Bandini, S. (2002). Integrating awareness in cooperative applications through the reaction-diffusion metaphor. Computer Supported Cooperative Work, 11(3–4), 495530.
  • Twidale, M.B., & Nichols, D.M. (1996). Collaborative browsing and visualisation of the search process. Proceedings of Aslib, 48, 177182. Retrieved June 10, 2010, from http://www.comp.lancs.ac.uk/computing/research/cseg/projects/ariadne/docs/elvira96.html
  • Twidale, M.B., Nichols, D.M., & Paice, C.D. (1995). Supporting collaborative learning during information searching. In Proceedings of Computer Supported Collaborative Learning (pp. 367374). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
  • Twidale, M.B., Nichols, D.M., & Paice, C.D. (1997). Browsing is a collaborative process. Information Processing & Management, 33(6), 761783.