SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Project
  5. Applying 3D interfaces in education: Art History
  6. Architecture
  7. Similarity measures
  8. Summary and Future Work
  9. Literature

This paper describes partially the architecture and the salient characteristics of a larger visual information retrieval (IR) project for Humanities purposes. The system uses a novel form of interaction for retrieval, sustainable, interactive exploration of retrieval sets, and the types of relationships that are germane to the use of the images, beyond what IR systems typically supply. There are gaps between the expression of a query and the retrieval set and the end-user's ability to establish significance for the set members. The paper sketches the architecture, interaction, and collection metadata created in response to a survey of art history student and instructors' educational needs.

It is argued that a 3D model provides visually-oriented people with a way to comprehend large datasets and, through the interactive control panel and additional graphics, identify facets of value by experts in the field and for local needs.


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Project
  5. Applying 3D interfaces in education: Art History
  6. Architecture
  7. Similarity measures
  8. Summary and Future Work
  9. Literature

As the volume of data grows, computer technology becomes more powerful and simpler, and end-user sophistication increases, the question of what information retrieval systems (IR) ought to supply end-users is altered. Should IR systems be used solely to locate objects or should they be tailored to a broader conception of user groups' information needs? More information about the records in an IR system might be used to help end-users both to locate and to interpret the data in a more direct way. For example, searching for visual resources (images representing 2Dand 3D art objects), end-users see the representation of the art, but without more knowledge cannot see the value of an object in society. 3D representations of retrieval sets capitalize on the visual orientation of the user and of the objects in a collection. In some domains, such as biology and medicine that rely on scientific visualizations, users expect a tighter integration of the visual representation of information and the supporting literature (Bergeron, 2002; Benoit, 2004). How might a visual emphasis for general users or other specific domain users make existing collections more useful?

This paper describes a project that intended originally to create a 3D interactive interface that could overlay existing collections to help a general user population search records. It became clear during an early Joint Application Development session that the concept of a “better” system was hollow because end-users were not interested in performing to traditional IR measures, such as searches faster, and wanted a system that supported their real goals, to learn more about a topic. In light of this, described below, the project changed to a second phase, focusing on a user population that is visual, but not scientific - art historians.

Metadata records for visual materials follow the Visual Resource Association's Core, version 4.0. VRA Core is an XML implementation that provides the usual descriptive and subject access, such as artist name, title, and descriptors from LCSH and the Art and Architecture Thesaurus (AAT), in other words, the “facts” of the object being described. But in general visual information retrieval (VisIR) systems focus either on searching by properties of the image itself (such as color, texture, or guessing at the shape) or operate as full-text systems that display sets of pictures instead of text as references to items in the collection. These efforts lack the integration of expert knowledge such that the metadata record contributes to the information seeker's understanding of the value of the object; there is a kind of fact/value dichotomy. Fortunately, XML is pliant enough to address this.

What is needed, perhaps, is a VisIR system that effectively permits and encourages domain experts to enrich the record by storing contextualizing data that reflect the values that help one understand the images. This requires a VisIR system that is designed to use the data, to supplement the IR processes and complement what the viewers extract from their information seeking session. The interface, of course, must support the user where these richer records meet their want of interpretation and knowledge. Unlike other VisIR applications (a) this project's search engine uses an innovative resource file of fuzzy associations between concepts in art history and (b)expanded metadata records that capture the values and interpretive perspectives that art educators value.

The rest of this paper describes how the project started, why it was altered, and how the mechanics of the redirected 3D interface work. It closes with some conclusions and plans for future work.

Related work

There are long-standing concerns in IR related to mapping queries to document collection representations and the source of mismatches, such as the semantics of query representation and documents' metadata contents. A review of the literature suggests there are identifiable streams of responses in visual information retrieval (VisIR) systems research. The first pursues the traditional IR approach of improving query/collection matching algorithms. The second focuses on the interface and the affective or emotional responses evoked.

The following two projects, as do many described in Ware (2004), layout a variety of affective measures for evaluating facets of VisIR systems. For example, Haik et al. (2002) focus on measuring task performance and user attitudes towards the display by enumerating what the researchers believe the end-users value: Navigation:

Table  . 
Navigation:  
Navigation:DifficultEasy
Mouse Use:DifficultEasy
Task:DifficultEasy
Orientation:LostOriented
Finding items:DifficultEasy
Freedom:FreeConstrained
Preference:DislikeLike
Time:LongShort
Remembering:SuccessfullyUnsuccessfully
Display:  
Display:Too denseNot dense
Confidence:Not confidentConfident
Effectiveness:Weak senseStrong sense
Control:Not in controlIn control of session

Chen & Czerwinski ([n.d]) consider user satisfaction, the speed of retrieval, and add spatial ability and the sense of immediacy of using the system in a suite of qualitative evaluative measures: Design satisfaction ratings for the user interface:

Table  . 
Usability
“The purpose of software immediately clear”
“It was easy to get what I wanted”
“I knew what to do”
“Each area of the software was clearly marked to indicate my location” Usability satisfaction ratings of the interface
simplicity, ability to zoom and walk around topics, navigation topical clusters Online Appeal:
software feels unique or different
mentally challenging
appealing graphics
responsive (not too slow)
provides valuable information
easy to use
“cutting edge” technology used
software provided a detailed environment to interact with
software is timely
personalized or customizable
shared experience (community)
feels familiar

These papers demonstrate the aesthetic attraction of visual IR, but no widely adopted evaluative criteria dominates (Morse, Lewis, & Olsen, 2002) to provide empirical results applicable to a recognized user group needs.

Moreover, they imply that affective measures are useful predictors of VisIR success, but this is not substantiated. Some groups (e.g., biologists, scientists in general; Benoit, 2002, 2004) are less swayed by emotional considerations, being more willing to endure uncertainty and complexity in order to explore hidden properties of the data in contextualizing retrieval results (van der Eijk et al., 2004).

This leads to a considerable and growing body of work, mostly proceedings, about individual visualization projects, explaining the designers' motivation and system architecture, without detailed empirical components. For example, Santini and Jain (2000) created “El Niño” as a suite of tools addressing what they term the “semantic gap” between query and searching visual collections. Santini, Gupta and Jain (1999) discuss the semantics emphasizing features, such as image color, structure and texture, emphasizing that the search “engine must be able to understand the placement of images in the display space … [and] be able to create a similarity criterion ‘on the fly’. … (p. 8). Urban and Jose (2006)describe “Ego: a personalized multimedia management tool” that also “should address the designing of a system that supports a variety of interactions and personalizing the support of information interaction” (p. 1). They create a product to support “retrieval in context”, learning from the user's personal organization. Others, Kang & Shneiderman's PhotoFinder (2000), Shen, Lesh, Vernier, Forlines & Frost's “Personal Digital Historian” (2002),collectively support the idea that visual retrieval is bound either to the text in the metadata record accompanying an image file or by some kind of automatic classification based on the image's color, structure, and texture. Borland 2003) also argues for more flexible modes of evaluating interactive information retrieval. She believes a “set of components” should be identified:

  • the involvement of potential users as test persons;

  • the application of individual and potentially dynamic information need interpretations deriving from e.g., the sub-component of a simulated work task situation; and

  • the assignment of multidimensional and dynamic relevance judgments.”

Recent work (Giereth et al., 2007; Spoerri, 2007; Luboschik & Schmann, 2007; Lau & Vande Moere, 2007) also value aesthetics, interactivity, and end-user understanding. The literature in general supports Chen's list of unsolved problems (2005): usability, understanding elementary perceptual-cognitive tasks, the impact of prior knowledge, eduation and training, quality measures, scalability, aesthetics, shifting to dynamic interfaces, visual inference, and knowledge domain visualization. Chen and Geroimenko advance the question by integrating XML (2005)and the Web (2006) into the challenge of making IV more useful.

From 2D to 3D:

Some researchers appeal to3D interfaces to address the issue of volume, similarity, and navigation (Cockburn& McKenzie, 2001; Wiza, Walczak& Cellary, 2004; Benoit, 2004). 3D has long been considered a way to address large sets of documents: “interactive 3D graphics techniques can be used to help the user comprehend and filter such [large]result sets” (Cugini, Piatko,and Laskowski, 1996, p. 1). Mackinlay,Robertson and Card (1992) and Calitz and Munro (2001)recognized the usefulness of visualization for hierarchical relationships. These and other studies examine how end-user spatial reasoning nad memory affect the user's interaction in full-text retrieval (Hemmje, 1993,1995). Newby (2002, p. 50) drew interesting conclusions from his Yavi system. He proposes that 3D interfaces

  • limit rotation, especially about the y axis, to lessen disorientation;

  • provide visual reference points (backgrounds, terrain, etc) to help enhance depth perception;

  • enable viewing of relationships beyond similarity (e.g., by selecting a document and seeing all terms in it);

  • de-emphasizing keystroke commands in favor of on-screen heads-up menus or pull-down menus, and

  • zoom and expanding/exploding the space.

He explains that this type of visualization is “more suited for visualizing several hundred, or perhaps a few thousand, items” (p. 49).

Eidenberger and Breitendeder's(2003) application “VizIR” integrated a 3D interface of icons but on a single plane, items receding from the user, similar to a stage, using a head-up display (“media panel”). This application parses XMLrecords to provide the data. Risden et al. (2004) go a step further examining 2D and 3D information visualizations integrating XMLand 3D (XML3D) but using colored balls with edges, superimposed over a globe with longitude and latitude lines.

Project

  1. Top of page
  2. Abstract
  3. Introduction
  4. Project
  5. Applying 3D interfaces in education: Art History
  6. Architecture
  7. Similarity measures
  8. Summary and Future Work
  9. Literature

Perhaps, though, 3Dinterfaces could be considered on smaller scales, with icons and interaction that are not [too] foreign to the users and tailored to identifiable user group interests. The interfaces of most 3D projects are attractive, at times visually stunning (e.g., NIRVE and Caida projects) but may be too centered on the research lab and too exotic for the typical information seeker (Benford et al., 1999).

As the volume and variety of resources available through web-based information retrieval systems grow, so does the need to display a large number of potentially heterogeneous objects in ways that encourage the seeker to interact with the system in different ways. The major difference is a shift from relevancy-ranked, text-based items in a result set to clustering items, to reduce the information space and visualization of resources to a manageable subset, controlled by the user. Consequently, VisIR necessitates a shift in how end-users think and what they expect of the IR system: is it only to identify and possibly provide access to copies of resources or is it to engage in a novel, communications-oriented exchange through which the seeker learns more about the collection, the topic, related concepts, relationships between concepts and resources that might not otherwise be noticed and so on?

The second major difference is in how the interface is perceived. Perception refers both to the intake of visual objects in three dimensions and how visualization affects the end-user's sense of certainty and comfort using new interfaces.

By having end-users provide feedback on data available through the interface and by providing their own thoughts about “missing” data, it is possible to get a realistic image of3D interfaces in the user's mind and perhaps to extract candidate factors that affect adoption. The question moves to how much “intelligence” should the end-user provide during the interaction (Hammond 2002). The search for stability also means looking at uncertainty. Horvitz 1999) discusses how designers could embed representation, “inference, and learning under uncertainty more deeply into the fabric of computer systems and interfaces.”

It seems, then, that many3D research projects recognize the interplay of perception, changes in IVsystem design from a graphic alternative to hierarchical lists to supporting, yet somehow integrating, the user's reasoning skills to retrieve documents and to learn from the experience.

Phase 1

The original goal of this project was to create and to test according to traditional measures a 3Dinterface that could be grafted onto into existing general retrieval collections. The original collection consisted of five hundred metadata records and their image files harvested from Harvard University's Hollis collection. In response to subject searching the application plotted the three most commonly founded subject tracings, assigning one to each axis. The search engine assigned the highest similarity measure value to documents whose primary tracing matched the query exactly and plotted it along the appropriate axis. The secondary and tertiary tracings were assigned a value, based on the normalized frequencies of all subject tracings in the retrieval set. Because the purpose of this first phase was to build a theory about the approachability of 3Dinterfaces, the similarity measurement was not highly refined.

The application written in Java, using data stored in MySQL tables for efficient searching, XML metadata records for each item, and served via the Web using Apache Jakarta Tomcat servlet container, all running on an Apple iMac offered end-users the following:

  • query text field,

  • option to show axes labels, change the size of the icons (from 10 to 100 pixels), alter the size of the retrieval set (1 to 100 objects),

  • option to view the original object (metadata record and image) in a separate frame, and

  • the system's rationale for including an object in a retrieval set.

Following Risden et al., (2004) the test interface offered two types of displays: a clustering of realistic, 3D, shaded balls, which size suggested the number of documents and a set of realistic icons (.png)that reflect the format of the documents in the retrieval set.

Originally, the project queries end-users of a 3D interactive interface retrieval program, using realistic icons in an information space, to see whether a small set of interactive opportunities, part/whole view of the result set, types of metadata, and the IR system's rationale for including an object in the retrieval set, are useful for relevancy judgments, and some of what users themselves project onto the interface to help themselves interpret the whole.

Two “Joint Application Development” (JAD) sessions were held with fifty graduate students and librarians, divided into two focus groups (each group was80% female, group one's median age was 25, group two's median age was 44). Such a sample should not be construed as an empirical investigation into the interface's qualities but only a development activity, accepted in both systems analysis and IA. Each group performed the same search (e.g., for “Hamlet” because the test collection represented a wide semantic range related to the concept of “Hamlet” and because the collection had a variety of formats (videos of the play, still images, articles with images, music)).

The first set of focus group opinions is the same as one expects from full-text search engines. Subjects wanted added to the interface the ability to

  • 1.
    mark images/records that have already been seen (e.g., change the icon's color),
  • 2.
    mark images/records to be kept,
  • 3.
    mark images/records to be deleted,
  • 4.
    limit the search to specific document formats, and
  • 5.
    g“more like these” relevance feedback option. In addition, they wanted features one would also expect from a graphic-oriented display:
  • 6.
    label the x, y, and z axes,
  • 7.
    add a “shopping cart” feature,
  • 8.
    alter the size of the retrieval set (too many hits make too much visual noise). The group also appreciated or wanted to add the following features:
  • 9.
    see the rationale for an item's inclusion in the retrieval set,
  • 10.
    add their own labels to individual items,
  • 11.
    add additional search terms or other features to manipulate the retrieved set (not to create a new search) and show the results on other axes,
  • 13.
    add their own color overlays to sub-groups in the retrieval set display (not system supplied). The most surprising outcomes were that end-users:
  • 14.
       wanted the interface to emphasize the interpretation of the data rather than suggesting how they might interpret the set by viewing realistic icons,
  • 15.
       felt more patient in their searching because of the aesthetics of the display,
  • 16.
       felt the aesthetics of the display increased their desire to use the interface, both diminishing the need to measure retrieval success over time and task performance;
  • 17.
       too much “graphic realism”, or as a one person described it “too much whiz-bang,” undermines their trust in the system.

Only one respondent, the oldest participant and a school library teacher, wanted an option to see the retrieval set as a hierarchical list. No participant wanted to search by the image's structure (e.g., color, texture).

Phase 2

In response to the focus groups' opinions, the application was redesigned:

  • shifting from icons that represent the format to the image itself [17],

  • providing a way to label and describe individual images [10],

  • build their own collection (“shopping cart”) [7],

  • altered the manipulation of the retrieval set [14, 11, 13, 8] through a “control panel”

The desire to help interpret the data lead to phase 2's shift to using Art and History students as a target audience. The rationale for this decision comes from need, expressed above, to create a reasonable model for evaluation that is not bound to task completion or finding records measured over time and which evaluation is pegged to a real user group.

Applying 3D interfaces in education: Art History

  1. Top of page
  2. Abstract
  3. Introduction
  4. Project
  5. Applying 3D interfaces in education: Art History
  6. Architecture
  7. Similarity measures
  8. Summary and Future Work
  9. Literature

Increasingly, educators want to integrate digital library contents into their courses by somehow capturing both the knowledge of the expert to help interpret the data representation and integrate local use needs. In addition, instructors want to expand the students' ability to contextualize images in the domain, here art history.

To this end, a third focus group of five participants was conducted using only Art History students and faculty to identify some of the values that are not addressed for that domain by VisIR systems. As one participant described it, “Comparing this to texts, say I'm interested in the ‘descent into madness’. I would have to know the scene in Shakespeare where Lady Macbeth says ‘Out, cursèd spot’ - there's no way I could search for what I want and get it. It's the same for images.”

Another participant said (referring to Velázquez's Las Meninas), “there's no way a student is going to know that the illumination of the frame's edge was actually inspired by Rubens and the mirroring similar to van Eyck unless they have studied it.”

In general, the group wanted an image retrieval system to address the needs of retrieval but also reflect their educational values, in a phrase to merge fact and value:

  • artist name,

  • subject of the work,

  • title of the work,

  • the era of the work's production,

  • patron of the work,

  • what did this item mean for the contemporaries of the artist?,

  • what does it mean for today's viewer?,

  • specifics of the work (e.g., self-portrait),

  • references in the work itself (e.g., a painting that references another painting in the work),

  • style or genre,

  • inspirations upon the artist,

  • the artist's inspirations on others,

  • specific facets of the work that make it noteworthy, and

  • a record of the use of the image in different classes.

thumbnail image

Figure 1. Las Meninas

Download figure to PowerPoint

The art historian's value system for retrieval is far more than just identifying the work for quick retrieval, and calls for a greater integration of working with the collection (a) to aid interpretation of a work of art's value and (b) see influences or other properties of images that are not otherwise possible in image retrieval systems.1

To address this, the system architecture was redesigned and a new collection created.

Architecture

  1. Top of page
  2. Abstract
  3. Introduction
  4. Project
  5. Applying 3D interfaces in education: Art History
  6. Architecture
  7. Similarity measures
  8. Summary and Future Work
  9. Literature

The architecture of the VisIR system is simple and scaleable. The below figure describes the life cycle of a query. The resource file of fuzzy weights of related concepts is used to alter the load on the axes, which is functionally equivalent to relevancy ranking.

thumbnail image

Figure 2. overview of system

Download figure to PowerPoint

Similarity measures

  1. Top of page
  2. Abstract
  3. Introduction
  4. Project
  5. Applying 3D interfaces in education: Art History
  6. Architecture
  7. Similarity measures
  8. Summary and Future Work
  9. Literature

The collection of objects consists of a XML metadata record, conforming to VRACore 4.0, using AAT, TGN,and ULAN for subject, geographic locations, and personal names, respectively.2

The second phase limited searching to subject fields. But to suggest relationships between the images that are not binary relations, such as other artistic movements, themes, etc.,a list of concepts (the educational values) was identified by an art historian, using Grove's Encyclopedia of Art as the warrant, and a fuzzy relationship established between the items. The values thus created were stored in a text file as a resource for determining (a) links between records and (b)relevancy ranking on three axes. Impressionism is related to other movements: post-impressionism, expressionism, abstract expressionism, and symbolism, e.g.:

Impressionism: {post-impressionism, expressionism, abstract expressionism, symbolism} 1: {.7, .5, .5, .5}

Retrieval set

The default size of the retrieval set is 25 items, but the user can adjust this. For each item in the retrieval set, a matrix of 3 rows [i.e., the subjects] x 25 columns [the digital records] is constructed. For instance, using only two matrixes a and b reflect the main subjects (rows) and then three specific objects' similarity measure for the given subject. The relevancy set for the entire collection, C, is the cross product of the objects that provides the values for loading the x, y, and z vertices(v): e.g., c23 = (a21, a22,a23)(b13, b23,b33) = (a21b13 + a22b23 + a23b33:

  • equation image

These values are used in a retrieval set file that feeds the data to the visualization applet as a set of parameters: and so on. 

Table  . 
Recordx-valuey-valuez-value
1100
2010
   
10.511
11.811782-.58396-1.2
121.1365-.817544-.12
13.811782-.58396-.2
   

Through the control panel, the end-user can manipulate the set on the three axes. The rotation of the objects follows:

  • equation image

3

thumbnail image

Figure 3. determining the vectors.

Download figure to PowerPoint

The results are presented to the user with the following options (Figure 4):

  • control panel to manipulate the search results without creating a new query,

  • select two pre-determined “educational values”,

  • alter the weight these items exert in the matrix,

  • show position labels (more relevant, less relevant),

  • show x, y, z axes,

  • show individual record numbers,

  • alter the size of the retrieval set,

  • alter the size of the images,

  • add a note to a selected image,

  • see the original source metadata record,

  • save a selected record to the user's own collection, and

  • delete a record.

Linking across clusters

In addition, while the 3Dclustering creates a representation of the retrieval set, it is useful to identify when some concept reaches across clusters. For example, given a retrieval set of three main themes (impressionism, symbolism, post-impressionism), say an artist's work in the collection is somewhat inclusive of impressionist works but a majority are classified under symbolism. Individual works (based on other data - the “educational values”) or groups of his works can be pinpointed in the retrieval set and a line drawn to link the individual item(s) of interest. The lines are drawn in 2D.

For the first object's points P1(x1, y1, z1) + the second object's P2(x2, y2, z2), we have l = cos α = (x1-x1/d) and m = cos β (y2-y1)/d, where d is the distance between P1 and P2: cos2 α + cos2 γ + cos2 γ = 1 or l2 + m2 + n2 = 1. The program's data structure records the screen location of individual items identifiable by a triple. As an example, say there are two items that are linked. Their locations are known by their points on the three axes: e.g., P1(2, -3, -3) and P2 = (-1, 1, 4). From this, it is a simple calculation to determine the end-points of the line that will link them. The distance equation image; x = z + lt, = -3 + mt, z = 3 + nt, where t runs from ∞to −∞ and the line segment of interest is [t1, t2]. [Naturally, the line segment is truncated for display.]

thumbnail image

Figure 4. Determining the line linking related visual items.

Download figure to PowerPoint

Identifying the beginning and end points, we paint a line between them, linking an object across clusters, as demonstrated in this example of works by John Singleton Copley:5

thumbnail image

Figure 5. The program in action: visual display results for a search of John Singleton Copley.

Download figure to PowerPoint

Summary and Future Work

  1. Top of page
  2. Abstract
  3. Introduction
  4. Project
  5. Applying 3D interfaces in education: Art History
  6. Architecture
  7. Similarity measures
  8. Summary and Future Work
  9. Literature

The purpose of the project was originally to create a 3D interface module based on VisIR literature and apply it to use existing records from an online catalogue. But the search for evaluating the interface beyond affective measures and beyond measuring the speed of retrieval or the success in locating an item, typical standards in IR, the preliminary phase of the project used focus groups to identify interactive behaviors and then progressed to the domain-specific needs of one visually-oriented group, art history. Consequently, the interface was redesigned and evaluation moved to supporting the information retrieval and educational values of art history education.

The project suggests that integrating local needs into metadata records that also reflect professional and domain standards (such as VRA Core) is probably worth the investment in human resources, but this has yet to be investigated.

The above project prompted the development of a new media-oriented, integrated web-enabled digital objects repository system, “auroraDL” (http://gslis.simmons.edu:8080/auroraDL), to realize end-user goals of more powerful, easier, and effective use of digital media in humanities and scientific education. This project is a collaboration between universities, museums, and research libraries to fulfill their several missions and to provide empirical data about actual information behaviors and interaction of end-users who work as they want to. The motivation is based on phase 2's focus group, during which subjects were encouraged to engage in “blue-sky thinking” about their needs. At first they felt dominated by technology and computing support offices, fearing to ask for anything. After being encouraged to design screens and to think about what they want, subjects actually became angry that their interests and success as educators were “dictated to by Technology” and felt forced to use ArtSTOR and other applications that, they claim, required them to change their work style. All participants complained about time “the waste being unable to share [their] own collections” more efficiently, unable to integrate their comments as experts (as one subject said “to enrich records and store the history of our using an image”) with the metadata records and, finally, felt the need to improve humanities education in general by reaching across subject field barriers. The strongest expressed user desire is the ability to create personalized, education-oriented digital collections on-the-fly from small-scale collections with quality images and records, rather than using large-scale, already existing collections that are costly to subscribe to and cannot be easily integrated into classwork.AuroraDL project with tailorable interfaces, collection building tools, and other education-oriented interactions, will implement the art historians' values and be used to study whether learning outcomes can be improved through interactive visual retrieval.

Literature

  1. Top of page
  2. Abstract
  3. Introduction
  4. Project
  5. Applying 3D interfaces in education: Art History
  6. Architecture
  7. Similarity measures
  8. Summary and Future Work
  9. Literature
  • Benoit, G. (2002, Aug). Data discretization for novel information retrieval. Journal of the American Society for Information Science and Technology, 53 (9), 736746.
  • Benoit, G. (2004, April). Properties-based retrieval and user decision states: user control and behavior modeling. Journal of the American Society for Information Science and Technology, 55 (6), 499497.
  • Benoit, G. (2004). Bioinformatics. In B.Cronin (Ed.) Annual review of information science and technology. Vol. 39. Medford, NY: Information Today.
  • Bergeron, B. (2002). Bioinformatics computing. New York: Prentice Hall.
  • Borland, P. (2003). The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Information Research, 8 (3), paper, no. 152.
  • Chen, C. (2005, July/Aug). Top 10 unsolved information visualization problems. IEEE Computer Graphics and Applications, pp. 1216.
  • Chen, C., & Geroimenko, V., (Eds.) (2005). Visualizing information using SVG and X3D. New York: Springer.
  • Chen, C., & Geroimenko, V. (2006). Visualizing the semantic web: XML-based Internet and information visualization. New York: Springer.
  • Chen, C., & Czerwinski, M. [n.d]. Spatial ability and visual navigation: an empirical study. Retrieved January 19, 2008 from http://research.microsoft.com/users/marycz/nrhm.htm
  • Cockburn, A., & McKenzie, B. (2001). 3D or not 3D? Evaluating the effect of the third dimension in a document management system. SIGCHI'01.
  • Cugini, J., Piatko, C., & Laskowski, S. (1996). Interactive 3d visualization for document retrieval. Retrieved July 06, 2006 from http://zing.ncsl.nist.gov/∼cugini/uicd/viz.html
  • Eidenberger, H., & Breiteneder, C. (2003). VizIR – a framework for visual information retrieval. JVLC.
  • Giereth, M. et al. (2007). Web based visual exploration of patient information. IEEE IV07.
  • Haik, E., Barker, T., Sapsford, J., & Trainis, S. (2002). Investigation into effective navigation in desktop visual interfaces. Web3D'02, pp. 5966.
  • Hemmje, M. (1993). A 3D based user interface for information retrieval systems.
  • Hemmje, M. (1995). LyberWorld – a 3d graphical user interface for fulltext retrieval. Retrieved July 06, 2006 from http://www.acm.org/sigchi/chi95/proceedings/videos/mhe_bdy.htm
  • Horvitz, E. (1999). Uncertainty, intelligence, and interaction. Proceedings of CHI'99, ACM SIGCHI Conference on Human Factors in Compuing Systems. New York: ACM, pp. 159166.
  • Kang, H., & Shneiderman, B. (2000). Visualization methods for personal photo collection: browsing and searching. IEEE International Conference on Multimedia and Expo III, pp. 15391542.
  • Luboschik, M., & Schumann, H. (2007). Explore to explain – illustrative information visualization. IEEE IV07.
  • Morse, E., Lewis, M., & Olsen, K. A. (2002). Testing visual information retrieval methodologies case study: comparing analysis of textual, icon, graphical, and ”spring” displays. Journal of the American Society for Information Science and Technology, 53 (1): 2840.
  • Newby, G. B. (2002). Empirical study of a 3D visualization for information retrieval tasks. Journal of Intelligent Information Systems, 18, 3153.
  • Norman, D. (2002). Complexity versus difficulty: where should the intelligence be. Plenary address, International Conference on Intelligent User Interfaces. http://www.iuiconf.org/02program.html
  • Risden, K., Czerwinski, M. P., Munzner, T., & Cook, D. B. (2004). An initial examination of ease of use for 2D and 3D information visualizations of web content.
  • Robertson, G., Czerwinski, M., Larson, K., Robbins, D. C., Thiel, D., & van Dantzich, M. (1998). Data mountain: using spatial memory for document management. Proceedings of the ACM UIST '98 Symposium on User Interface Software & Technology, pp. 153162.
  • Santini, S., & Jain, R. (2000). Integrated browsing and querying for image databases. IEEE MultiMedia, 7 (3), 2639.
  • Santini, S., Gupta, A., & Jain, R. (1999). A user interface for emergent semantics in image databases. Proceedings of the 8th IFIP Working Conference on Database Semantics (DS-8). http://citeseer.ist.psu.edu/58533.html
  • Shen, C., Lesh, N. B., Vernier, F., Forlines, C., & Frost, J. (2002). Sharing and building digital group histories. Proceedings of the 2002 ACM conference on CSCW, New Orleans, LA. New York: ACM, pp. 324333.
  • Spoerri, A. (2007). Visual mashup of text and media search results. IEEE IV07.
  • Urban, J., & Jose, J. M. (2006). EGO: a personalized multimedia management tool. International Journal on Intelligent Systems, 21 (7), 72545.
  • Wiza, W., Walczak, K., & Cellary, W. (2004). Periscope – a system for adaptive 3d visualization of search results. Proceedings of the 9th International Conference on 3D Web Technology. Monterey, Ca. New York: ACM, pp. 2940.
  • Ware, C. (2004). Information visualization. 2nd ed. San Francisco: Morgan Kaufmann.