Many visualization systems use brushing, selections and linked views to provide means for the exploration of complex data sets. There are various tools for specifying the selections and they usually serve only one specific purpose. The idea of contextual snapshots is to harness a single mechanism of 2D spatial selections for different tasks, such as data selection and manipulation, data annotation or specification of DOI functions. In contextual snapshots, this is achieved by the following concepts: multiple selections can be stored and each of them can be created in a different context (i.e. parameter settings of the visualization system); an algorithm of transforming the user input to the DOI functions of the selections is interchangeable; each selection can be linked with a number of additional views which we refer to as embedded visualizations. It is possible to display, that is, embed them, directly in the visualization image. Embedded visualizations are interactive and they can display arbitrary graphical user interface elements or visualize data specified by a corresponding selection.
In the example application, the selections can specify a spatial region in image space of a volume visualization. An embedded view which displays the histogram of the volume data in the specified region is linked with each selection. Contextual snapshots calculate a histogram of the selected voxels and provide it to the embedded visualization. How these data are used depends on the implementation of individual embedded visualizations. Another embedded visualization is a simple text field. It does not display the selected data region, but it allows users to type in arbitrary text-based annotations. Such an embedded visualization is linked with every selection, thus providing a possibility to annotate selected data subsets.
3.1. Concept of contextual snapshots
A visualization system may apply various parameters to modify the visual mapping. With respect to contextual snapshots, the values of a chosen subset of said parameters, the visualization snapshot, define a state of the visualization system. For any state of the system, a user can create several selections in the rendered images. A contextual snapshot stores a visualization snapshot together with all selections created when the state of the visualization system corresponded to this visualization snapshot.
In the hurricane visualization application, we use the position and the orientation of the virtual 3D camera as the parameters stored within the visualization snapshot. Therefore, each image–space selection is bound to one 3D camera view. The selection is valid only if the volume is rendered using this camera view. Other parameters, such as current timestep, are not stored in the visualization snapshots. Changing these parameters does not make the selections invalid. This is application-specific and in the given case, it allows users to explore how the hurricane data change over time in specified spatial regions. When contextual snapshots are integrated with a visualization system, the system integrator has to choose the set of parameters which appropriately define the context for the selections.
A contextual snapshot is defined as:
The idea of contextual snapshots is based on the fact that the semantics of the user-made selections depend on what is currently displayed. When the user selects a particular feature in the image, the selection is meaningful only until the way how the feature is displayed changes. Therefore, we extract a visualization snapshot every time a new selection is created. The visualization snapshot is linked to the selection to create a contextual snapshot. The contextual snapshot then provides a reproducible spatial selection related to what was displayed when the selection was made. It stores the appropriate visualization context in the form of the values of the visualization-system parameters. All selections with the same visualization snapshot are stored together within one contextual snapshot.
The strength of contextual snapshots is that they can maintain multiple selections created in different states of the visualization system. The information stored within a contextual snapshot can be used to restore the given state, so that the selections can be displayed and actively used. By restoring the state of the visualization system according to the individual contextual snapshots, it is possible to browse all selections created within a visualization session.
Contextual snapshots are represented by icons which we refer to as anchors. The anchors are embedded in the original visualization as interactive graphical elements. They constitute abstract previews of the corresponding contextual snapshots. For instance, in the hurricane visualization application, the anchors are positioned in 3D space to represent the camera positions when the respective contextual snapshots have been recorded. An anchor can also display a thumbnail of how the visualization looked like when the respective contextual snapshots have been created. The anchors are interactive and they are used to restore the visualization-system state to the respective contextual snapshot. They serve as a user interface for browsing through the contextual snapshots created during the visualization session. Figure 1 shows the graphical representation of anchors.
Figure 1. Interactive anchors (white circles) representing individual contextual snapshots. For better 3D orientation, the anchors are connected with the coordinate origin by a thin line; (a) shows how the thumbnail of an anchor can be displayed; (b) and (c) show the anchors from different camera views.
Download figure to PowerPoint
The visualization-system parameters used as context for the selections vary in different applications. The graphical representation of anchors can be customized to convey the information stored in the contextual snapshot, as demonstrated in Section 'Application Example: Heart Visualization'. The default implementation uses the viewing–transformation matrix of the camera to position the anchors in 3D space. In this case, the position of the anchor conveys the position of the camera at the time when the contextual snapshot has been recorded. This is only suitable for visualizations using a 3D camera. For other use cases, we allow programmers to set an arbitrary screen–space position for each anchor. This way, the anchors can be positioned with respect to a feature of the visualization to potentially reveal the semantics or content of the contextual snapshot. This is demonstrated in the use case described in Section 'Application Example: Historical Document Analysis'.
3.2. Embedded visualizations as linked views
To broaden the possibilities of using context-aware selections, we provide a method for linking interactive embedded visualizations for each selection. The additional visualizations can show different aspects of the selected data, or they can display comparisons of various selected areas. To demonstrate different ways how the selections can be used, we implemented the following embedded visualizations for the hurricane visualization application: a histogram of selected data values, a text-based annotation widget and a variable picker.
The embedded visualization displaying the histogram of selected data values can be linked to multiple selections at once. It shows histograms for individual selections in overlays so that they can be easily compared. A separate text-based annotation widget is linked to each selection. It provides the user with the possibility to type in a short description of the selected data subset. The variable picker is a graphical user interface element which displays a list of all variables present in the data set. The chosen variable is displayed in those parts of the image, where the selection has been created. Figure 2 shows how the picked variable is integrated with the rest of the visualization by a smooth transition. The smooth transition is due to the non-binary DOI function of the selection.
Figure 2. (a) A selection (marked with the red circle). (b) An integrated view of two variables using the selection after its activation. Histograms are shown for both variables as embedded visualizations. The third embedded visualization is a variable picker—a list of the data variables, where the user can choose which one is displayed.
Download figure to PowerPoint
In our method, each image–space selection can be linked with multiple embedded visualizations. Each embedded visualization has access to the data subsets specified by the selections to which they are linked. A single embedded visualization can be used to display and compare aspects of different subsets of the explored data by simply linking it with multiple spatial selections.
For the purpose of displaying the embedded visualizations, we propose a mechanism for activating individual selections. One or multiple selections can be activated by the user at once. In this case, only those embedded visualizations are displayed which are linked with every activated selection. The rationale for this mechanism is that the embedded visualizations can show different aspects of the data specified by multiple selections at once. This way, the selections and their embedded visualizations can be used to compare several data subsets.
A sketch-based interface is used for activating selections. The user activates selections by painting a stroke in image space. The selections which are crossed by the stroke are activated and subsequently their linked embedded visualizations are displayed. The embedded visualizations are grouped together in a sliding bar, which is displayed either at the top, the bottom, the left or the right side of the visualization image. The position of the sliding bar is determined by the direction at the end of the stroke which was used to activate the selections. The interaction method of using a stroke was chosen so that an arbitrary subset of the selections can be activated, which might be difficult with various standard selection mechanisms.
The sliding bar is capable of showing several embedded visualizations at once. In case there are more embedded visualizations for the activated selections than actually fit on the screen, the sliding bar enables scrolling of its content. The scrolling is executed by an animated transition, so that the users have a visual feedback on the direction of the scrolling. The sliding bar fades to the original visualization on both sides for better integration. A gradual blurring filter was used on both sides of the sliding bar, so that the attention of the users is guided to the embedded visualizations currently shown in the middle of the bar.
Displaying several embedded visualizations at once provides users with an overview of the additional data depicted for the activated selections. However, to facilitate interaction it might sometimes be necessary to enlarge individual embedded visualizations. For this purpose, the embedded visualization displayed in the middle of the sliding bar can be switched to a so-called maximized view. If the view is maximized, the embedded visualization is displayed on the whole screen rather than just in the sliding bar.