Investigating the creation of ad hoc information visualizations

Authors


Abstract

While our frameworks for the analysis of textual information and verbal communication are rich, our means for performing the same investigations of visual information are comparatively weak. Given the increasingly visual orientation of our culture and technologies, this is problematic. This poster presents a case for the study of ad hoc visualizations as both communicative practice and information behavior. Ad hoc visualizations are images spontaneously created during conversations. Often corresponding to moments of heightened clarity or coordination, the creation of such visualizations can be viewed within a broad communicative context, alongside linguistic and other non-textual modes of exchange. Further study of image-enabled discourse, and specifically ad hoc visualizations, will contribute to the field of information science by extending existing research related to information behaviors into the realm of information creation. It will also provide a more inclusive framework for the study of visual information.

Problem statement

Ad hoc visualizations are images spontaneously created during conversations. Often corresponding to moments of heightened clarity or coordination, the creation of such visualizations can be viewed within a broad communicative context, alongside linguistic and other non-textual modes of exchange. Marks on a napkin or sketches created on a white board are information artifacts that embody a particular type of communication practice that plays a specific role in the exchange of meaning between individuals. Image-enabled discourse refers to this phenomenon in a broad sense, and ad hoc visualizations are one type of image-enabled practice. This poster presents a proposed study of ad hoc visualizations as both communicative practice and information behavior.

We traditionally think of information as primarily textual (Buckland, 1991), although advances in the creation and distribution of multimedia content has challenged the efficacy of this assumption. Previous research in the area of image retrieval highlights some of the problems inherent in attempting to apply text-based principles of information behavior to the visual domain (Brunskill & Jorgensen, 2002; Greisdorf & O'Connor, 2002a, 2002b; Jorgensen, 1998). In fact, our methods and frameworks for conducting investigations of visually oriented artifacts and behaviors are relatively weak when compared to the work devoted to text-based information. Information science does not yet have a truly comprehensive methodology for studying a range of phenomenon related to the creation and use of visual information. Given the increasingly visual orientation of our culture and technologies, this is problematic.

When focusing on the creation of images within the context of a conversation, visualization can be seen as an interactive communicative action or activity. The approach developed for this study is based in the notion that communicating through drawing is a unique discourse strategy that can serve specific roles within multimodal conversations. The choice to deploy an image, rather than to use another mode of communication such as spoken language or gesture, is itself an informative action, above and beyond the specific content being represented in a visual format. In order to gain a more comprehensive view of the ways that visual information can function in communication, it is helpful to devote attention to image-enabled discourse practices as a complement to more artifact-centered image research.

Research questions

This study builds on the notion that affordances provided by ad hoc visualization practices (i.e. drawing) create conditions salient or hospitable for using specific image-enabled conversation strategies. The research will address the following questions: What communication practices are associated with the creation and use of visual information in face-to-face interactions? What role does the creation of ad hoc visualizations play in face-to-face communication practices? Which affordances of visual modes of communication are most salient to the use of ad hoc visualization strategies?

Method

The dynamic and spontaneous nature of the creation of ad hoc information visualizations makes it challenging to study the phenomenon in a systematic manner. A natural and ideal scenario would be to gather observations “in the wild.” However, as easy as it is to think of instances in the past when images were created in the course of a conversation, it is challenging to predict when such moments will occur in the future. Therefore, a protocol will be designed for systematically capturing image-enabled interactions in a naturalistic, observable environment. The intent here is not to create a controlled experiment, but instead to create situations where observations can be made in a reliable and consistent manner while providing an authentic experience for participants.

Preliminary study

Participants in a preliminary study provided narratives of conversations in which they engaged where at least one drawing was created during the course of the interaction. Two research techniques commonly used in information science research guided the design of this exploratory study: critical incident technique (CIT) (Flanagan, 1954) and sense-making methodology (Dervin, 1999). During semi-structured interviews (delivered face-to-face and via a web-based questionnaire) respondents were asked to think of a specific face-to-face conversation involving one or two other people when a drawing was created during the course of the conversation. Inductive analysis of fifty-one transcripts revealed emerging patterns of use associated with the creation of ad hoc information visualizations.

Five preliminary practices associated with image-enabled discourse were identified:

  • Consensus – in which representations of a concept are created and normalized

  • Persuasion – in which one person seeks to re-focus the attention of another to more closely align with a specific ideal

  • Verification – in which the form of a message is changed in order to verify understanding

  • Visualization – in which specific information is conveyed in a mode as close to its original expression as possible

  • Synchronization – in which a waypoint(s) is created that marks synchronized understanding, before moving forward

In addition, five enabling affordances were identified in the preliminary study:

  • Plasticity – the ability to simultaneously make reference to different points in space and time

  • Mutually accessibility – the ability for all participants to make formative contributions based on access

  • Symbolic nature – the ability for images to support and augment other semiotic systems in use

  • Authoritativeness – the ability that visual representations have to lend credence or authenticity to a concept

  • Visuality – the self-evident ability for images to clearly and accurately represent visual information

While these categorizations are certainly nascent, the proposed study will provide further evidence of practices and affordances as well as identify any additional attributes exploited during the deployment of image-enabled communication strategies. A further objective of the proposed study will be to determine the relative importance each of the affordances has for the identified visualization practices.

Proposed study protocol

For the main study, participants will be placed in dyads and asked to complete a problem task as a team. In order to preserve the naturalistic aspect of image-enabled behaviors, the fact that the study is focused on the creation of images will not be shared with participants. The problem will provide multiple opportunities for the use of any or all of the practices identified in the initial study, but will not close the door to the use of alternate image-enabled, and even non-image enabled, communication practices. It is expected that interactions will result in a range of practices and outcomes, some involving the creation of images and some not, allowing for comparisons across conversations where drawings occurred and those where it did not.

This study will look at image-enabled communication as a widely accessible strategy practiced by a broad range of people. While observation of highly sophisticated visualization practitioners may yield interesting heuristic results related to best practices, focusing on such a group would likely obfuscate the real objective of the research: to create a generalized view of visualization practices (good, bad and everything in between).

Overall performance will not be measured, as the purpose of the task is to set a duplicable stage for observation of communication between participants. The task used for the study will be determined through pilot testing. An example from cognitive psychology is provided to illustrate the type of prompt that will be used. Duncker's candle problem is frequently used as an experimental task in problem-solving research: Given a candle, a box of nails, a matchbook and a hammer, the problem is to get the candle up on the wall so that it burns properly. Any of the objects may be used (Weisberg & Weisberg, 1980, pp. 251-252).

Commonplace office supplies (such white board, paper and pens) will be provided, though no specific prompts will be given in order to elicit the use of images during tasks. The problem session will be followed by individual exit interviews very similar to the interview protocol used during the preliminary study.

Analysis of image-enabled conversations

The objectives of the proposed study are to create a comprehensive analytic scheme for identifying image-enabled communication practices and to identify the specific enabling affordances of visual information that contribute to the strategic deployment of these practices during face-to-face conversations. These objectives will guide multimodal interactional analysis of the video/audio recordings and textual analysis of interview transcripts.

The framework for the analysis of the image-enabled conversations will be influenced analytic frameworks adapted from the fields of discourse linguistics and multimodal communications. Discourse strategies (Gumperz, 1982) and multimodal interactional analysis (Kress & Leeuwen, 2001; Norris, 2004 are related areas of study that focus on the ways that the form and format of an utterance (not just the nominal content) can influence interpretation and inference within conversations. Underlying both of these approaches is the notion that disambiguation of meaning is an interactive process and all choices related to the expression and interpretation of meaning (both linguistic and nonlinguistic) made by participants in a conversation influence the construction of meaning during an exchange. Therefore, these choices are informative in and of themselves, augmenting what is being conveyed by the actual words spoken.

Both Gumperz and Norris pay particular attention to types of shifts that occur during conversations. In his work on bilingual communication, Gumperz refers to code shifts in multi-language conversations as not just evidence of deep and sometimes unconscious expressions of cultural values and norms (for example, using a native tongue to talk about family while shifting to an adopted language when talking about work) but also as evidence of strategic language practices used to support interactive inference and interpretation (for example, bilingual parents who reinforce the imperative of reprimands to their children by repeating a chastisement in two different languages).

For Norris, modality (such as speech, gesture, head position, printed material, or music) is an informative dimension of discourse that can explain various aspects of meaning and interpretation. Multimodal interactional analysis looks for shifts in the relative density of modes of communication in face-to-face conversations, as well as changes in the intensity of modes at different points in the conversation. His analysis of a simple conversation over the breakfast table would encompass not just the spoken language used by participants, but also body language, the layout of the kitchen, the presence of printed material (such as newspapers) with which participants might be independently engaging, and the narrative of a morning news program that can be heard from the next room. His interest is in how these modalities interact and influence the flow of communication between individuals.

The analytical framework for studying image-enabled discourse will be influenced by these shift-oriented perspectives on the construction of meaning in conversation. The concept of shifts will ground analysis by providing a structure for marking episodes of interaction between conversants as well as informing categorization of those episodes within an overall framework of image-enabled practices.

Contributions of the proposed study

Information science can benefit greatly from viewing the creation of ad hoc visual information in a similar social interactive context. By focusing on the creation of visual images as a unique, informative discourse practice analogous to uttering a word or making a gesture, we can produce more comprehensive analyses of image-enabled exchanges, enhancing our understanding of both visual information artifacts and practices. Richer analyses of image-enabled conversation could have implications for computer-supported collaborative work, as well as contributing to more robust, context specific document models for image retrieval.

Ancillary