SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. RELATED WORK
  5. RESEARCH METHODS
  6. RESULTS
  7. DISCUSSION
  8. CONCLUSIONS
  9. Acknowledgements
  10. REFERENCES

We investigate how demands of a task create differing task roles with distinct levels of information uncertainty, information and cognitive load, and social responsibility. Pairs solved a standard matching game using tangrams under three different instant messaging (IM) tool conditions. We show that a participant's use and evaluation of the IM tool are largely based on their assigned task roles. The findings inform the testing and evaluation of future collaborative systems and visualization tools by demonstrating the relevance of task roles in the use and utility of these systems and tools.


INTRODUCTION

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. RELATED WORK
  5. RESEARCH METHODS
  6. RESULTS
  7. DISCUSSION
  8. CONCLUSIONS
  9. Acknowledgements
  10. REFERENCES

The move from bounded collaborative clusters to large-scale, networked partnerships spanning regionally, nationally, and internationally is creating a need for the design of effective collaborative systems. One area that has received particular focus is the design of instant messaging (IM) tools that support interactivity (Nardi, Whittaker & Bradner, 2000), awareness (Bradner, Kellogg &Erickson, 1998), and presence information (Erickson & Kellogg, 2000; Quan-Haase, Cothrel & Wellman, 2005). Nonetheless, this area of research has largely neglected to investigate whether or not and how different communicators' task roles influence the perceived usability of a tool. An exception is Korkini et al.'s (2011) study on the effects of users' mental workload on perceived tool usability. Building on their work, we investigate the link between task roles and the use and perceived utility of collaborative systems. Our preliminary findings support this link, showing that users' perceived usability of an IM tool can vary depending on the information uncertainty, information and cognitive load, and social responsibility associated with their task roles.

RELATED WORK

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. RELATED WORK
  5. RESEARCH METHODS
  6. RESULTS
  7. DISCUSSION
  8. CONCLUSIONS
  9. Acknowledgements
  10. REFERENCES

IM provides an informal, spontaneous means of communication that allows collaborators to ask quick questions, to obtain prompt clarifications, and to coordinate meetings (Nardi, Whittaker & Bradner, 2000). Further advantages include the ability to negotiate social accessibility, to conduct intermittent conversations, and to maintain a sense of connection with others, even without necessarily communicating (Nardi, Whittaker & Bradner, 2000; Quan-Haase et al., 2005).

Media richness theory (Daft & Lengel, 1986) provides a lens for understanding the limitations of IM for collaboration. The theory proposes that rich media support the exchange of complex messages, while lean media are only adequate for unequivocal messages. To overcome the limitations of text-only lean media, various approaches have been suggested; e.g., to improve message ordering (Xiao, Litzinger & Unravel, 2005), to provide contextual information (Bradner, Kellogg & Erickson, 1998), to visualize communicators' rationales (Xiao, 2012), and to enhance awareness to prevent interruptions and distractions (Cutrell, Czerwinski & Horvitz, 2001).

The use of IM tools in various contexts has been studied extensively as well, e.g., in learning contexts (Wijekumar et al., 2005), in gaming environments, such as MMORPGs (Christopherson, 2010), and in virtual shared workspaces (Saparova et al., 2011). Prior studies have also looked at different variables that affect users' IM experiences, e.g., the effects of gender on trust perception and performance in CMC (Sun et al., 2007). In the past, the perceived usability of IM tools has been measured without differentiating between different communicators' task roles. This makes the assumption that other than individual differences, all communicators will have the same kind of feedback on the design and usability of IM tools. However, Quan-Haase et al.'s (2005) study has suggested that task roles impact the nature of message exchanges. They found that people's social status and social ties (close vs. distant) directly affected the use of IM for solving complex tasks. This motivated us to examine in the present study how different task roles as defined in a matching game (directors/matchers) could impact the use and perceived usability of IM tools.

RESEARCH METHODS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. RELATED WORK
  5. RESEARCH METHODS
  6. RESULTS
  7. DISCUSSION
  8. CONCLUSIONS
  9. Acknowledgements
  10. REFERENCES

Participants

University students who had prior experience with IM were recruited. A total of eighteen students participated in the study as 9 pairs. However, data from two pairs had to be discarded because of unexpected events during the experiment. Thus, we include 7 pairs: 7 male and 7 female participants. Twelve were native English speakers. Of the 14 participants, 10 indicated using IM tools “daily,” two indicated using it “rarely,” one indicated using it “once a week” and one indicated using it “a few times a week”. Thirteen of the 14 participants were in the age range of 19-25, and one participant chose not to reveal her age.

Tasks and Tools

Participants were given the matching task using tangram figures developed by Clark et al. (1986). In this task, one participant is assigned the ‘director’ role and his/her communication partner the ‘matcher’ role. In our task, directors were instructed to describe four tangram images consecutively to matchers via IM. Matchers were given a set of 16 images at the beginning of the study and they had to either identify the image from this set or make a note if the image was absent.

Two IM tools were used in the experiment: GTalk and Jitsi. Jitsi, is based on open source software developed in Java, supports both text-based chat and video streaming. In this experiment, video was not used. As the pairs communicated via IM, they were instructed to put asterisks (*) around words or phrases of importance; they were required to use asterisks at least 10 times under each tool condition. No time limit was set for sessions. In GTalk, the words/phrases between asterisks are highlighted in bold in the chat message window. Because the highlight feature was not available in Jitsi, we designed a Message Flower that visualizes user identified words/phrases in the form of a flower petal. The color of petals turned from yellow to green when participants clicked on it.

Two kinds of visualizations were used in the study: a) sender-centered Message Flower (see Figure 1) and b) image-centered Message Flower (see Figure 2). In the sender-centered Message Flower, the selected message portion was displayed around its sender, i.e., the director or the matcher. By contrast, in the image-centered Message Flower, the selected message was displayed around each of the four images. For the Jitsi tool condition with the Message Flower, three pairs were assigned to the sender-centered flower and four to the image-centered flower.

thumbnail image

Figure 1. Screenshot of a Sender-Centered Message Flower

Download figure to PowerPoint

Experimental Design

The experiment used a within-subjects design with three tool conditions: GTalk, Jitsi, and Jitsi with the Message Flower visualization. The order in which tasks and tool conditions were administered to pairs was randomized. In each tool condition, pairs were asked to perform one matching task. If a pair had correct matches for all three conditions, each participant received $25 for compensation. If the pair completed all three conditions but did not get all the matches correct, each participant received $10 for compensation.

thumbnail image

Figure 2. Screenshot of an Image-Centered Message Flower

Download figure to PowerPoint

Procedures

Upon arrival, participants were taken to two separate rooms where the experiment took place and they were provided the letter of information about the procedures.

After completion of each tool condition, participants were given an online survey which evaluated on 7 items the usability of the tool in terms of ease of use, effectiveness, and satisfaction on a 4-point Likert scale ranging from “strongly disagree” (-2), “disagree” (-1), “agree” (1), to “strongly agree” (2). Following the online survey, participants continued with the next condition. The survey also included items about the participant's background that is related to the task, e.g., “What is your first language?” and “How often do you use instant messaging tools?”

In total each participant took the same survey three times, one time following each tool condition. After completing all three conditions, participants took part in an in-person, semi-structured interview. The interview questions were designed to shed light on the participants' tool preference and to obtain insight into the reasons for their choices. All responses were recorded through note-taking.

RESULTS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. RELATED WORK
  5. RESEARCH METHODS
  6. RESULTS
  7. DISCUSSION
  8. CONCLUSIONS
  9. Acknowledgements
  10. REFERENCES

All pairs completed all tasks. Two pairs of seven had correct matches for all three tasks.

Communicators' Task Roles and Perceived Usability

As shown in Table 1, directors provided more positive feedback than matchers about the design and usability of the IM tools and this was observed across all tool conditions and for all survey items. Paired t-tests for directors and matchers' responses to item 7 showed statistical significance independent of tool conditions: for GTalk, t(6) = 2.75, p < .05; for Jitsi, t(6) = 2.70, p < .05; and for Jitsi with visualization, t(6) = 2.99, p < .05.

Table 1. Director and Matcher's responses on the survey items
Thumbnail image of

The interview data paralleled the findings from the online survey. Five of the six interviewed directors liked the Jitsi Message Flower condition the most (one director's interview data were lost during the data collection process). When asked for the reasons, directors responded that visualization helped organize thoughts, supplement chat sessions, and emphasize relevant points in the communication process. They also thought it was aesthetically pleasing. Although three matchers liked the visualization tool condition the most, the remaining four matchers considered the Message Flower to be distracting.

Preferences of Message Flower's Visualization Options

The interview data suggest that in this experiment the image-centered option (Figure 2) was preferred to the sender-centered option (Figure 1). For example, the only director who did not like the Message Flower tool had the sender-centered option during the experiment and he said it was “not useful and unnecessary”. Those matchers who liked the Message Flower tool most all had the image-centered option. Also three of the four matchers who liked the Message Flower least had the sender-centered option.

DISCUSSION

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. RELATED WORK
  5. RESEARCH METHODS
  6. RESULTS
  7. DISCUSSION
  8. CONCLUSIONS
  9. Acknowledgements
  10. REFERENCES

Our survey data show that five directors and five matchers use IM tools daily. One director uses IM tools a few times a week, and one once a week. Two matchers rarely use IM tools. Given this result, we consider that the participants' level of expertise would not play a significant role if we compare the responses of two groups, i.e., the director group and the matcher group.

We attribute the differences in the evaluation of the tools' usefulness between directors and matchers to the information needs and medium requirements that are associated with each task role. For directors, their primary assignment consisted of interpreting the information provided by the matcher via IM to determine if they had identified the correct image. By contrast, the task of the matcher consisted of interpreting the information from the director and additionally guessing and extrapolating based on this information. Therefore, even though both directors and matchers worked collaboratively to solve the problem, matchers had more information uncertainty in the task. As Choo (1999) demonstrated, those who perceive the environment to be uncertain tend to seek out more information.

In the experiment, the matchers experienced higher cognitive load than the directors because they were required to process information from various sources, which put higher demands on their attentional resources. For them the tool needs to not only help with information seeking, but also with the additional tasks of selecting, identifying, and extrapolating. Hence, the matcher may have evaluated the flower message visualization tool as distracting since it did not provide any additional benefits to the matcher's tasks. This explanation is consistent with Korkini et al.'s (2011) findings which demonstrated the impact of an individual's mental workload on perceived tool usability and may be particularly relevant in the context of groupware.

CONCLUSIONS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. RELATED WORK
  5. RESEARCH METHODS
  6. RESULTS
  7. DISCUSSION
  8. CONCLUSIONS
  9. Acknowledgements
  10. REFERENCES

Our preliminary study examines whether or not and how communicators' task roles affect the perceived usability of IM tools. Our preliminary findings suggest that task roles are associated with different information needs and medium requirements; and these affect the perceived usability of a tool. This finding has important implications for technology design because it proposes that task roles—specifically, the associated level of information uncertainty, information and cognitive load, and social responsibility—are important factors to be considered in both the design and evaluation of collaborative systems and visualization tools.

REFERENCES

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. RELATED WORK
  5. RESEARCH METHODS
  6. RESULTS
  7. DISCUSSION
  8. CONCLUSIONS
  9. Acknowledgements
  10. REFERENCES
  • Bradner, E., Kellogg, W. A. and Erickson, T. Babble. (1998). Supporting conversation in the workplace. SIGGROUP Bulletin, 19, 3, 810.
  • Christopherson, L. (2010), What are people really saying in World of Warcraft® chat? Proc. Am. Soc. Info. Sci. Tech., 47: 12. doi: 10.1002/meet.14504701340
  • Choo, C. W. (1999). The art of scanning the environment. American Society for Information Science, 25, 3, 2124.
  • Clark, H. H., and Wilkes-Gibbs, (1986). D. Referring as a collaborative process. Cognition, 22, 139.
  • Cutrell, E., Czerwinski, M., & Horvitz, E. (2001). Notification, disruption, and memory: Effects of messaging interruptions on memory and performance. In M. Hirose (Ed.), Human-Computer Interaction: INTERACT '01 (pp. 263269). Amsterdam: IOS Press.
  • Daft, R. L. and Lengel, (1986). R. H. Organizational information requirements, media richness and structural design. Management Science, 32, 5, 554571.
  • Erickson, T. and Kellogg, (2000). W. A. Social translucence: An approach to designing systems that support social processes. ACM Transactions on Computer-Human Interaction, 1, 1, 5983.
  • Lee, S., Koubek, R. J. and Moon, (2011). K. Considering context: The role of mental workload and operator control in users' perceptions of usability. Int. J. Of Human-Computer Interaction. DOI: 10.1080/10447318.2011.622973
  • Nardi, B. A., Whittaker, S. and Bradner, (2000). E. Interaction and outeraction: Instant messaging in action. In Proceedings of the 2000 ACM conference on Computer supported cooperative work (CSCW '00). ACM, New York, NY, USA, 7988.
  • Quan-Haase, A., Cothrel, J. and Wellman, (2005). B. Instant messaging for collaboration: A case study of a high-tech firm. J. of Computer-Mediated Communication, 10, 4. article 13. http://jcmc.indiana.edu/vo110/issue4/quan-haase.html
  • Saparova, D., Basic, J. and Kibaru, F. (2011), Exploring the usefulness of awareness widgets in online shared workspaces: A preliminary case study. Proc. Am. Soc. Info. Sci. Tech., 48: 14. doi: 10.1002/meet.2011.14504801301
  • Sun, X., Wiedenbeck, S., Chintakovid, T. and Zhang, Q. (2007), The effect of gender on trust perception and performance in computer-mediated virtual environments. Proc. Am. Soc. Info. Sci. Tech., 44: 114. doi: 10.1002/meet. 1450440211
  • Wijekumar, K. and Meidinger, P. (2005), Interrupted cognition in an undergraduate programming course. Proc. Am. Soc. Info. Sci. Tech., 42: n/a. doi: 10.1002/meet.l4504201168
  • Xiao, L., Litzinger, J., (2005). Unravel the ordering in persistent chat: A new message ordering feature, In Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work (GROUP '05). ACM, New York, NY, USA, 450451.
  • Xiao, L. (2012). Rationale Flower: a Visualization tool for identifying hidden profiles in instant messaging, the 2012 ACM Conf. on CSCW (February 11–15, Seattle, WA), 255258.