The paper presents the results of a comparison of two interfaces, one a conventional taxonomy of terms relating to Canadian history, and the other a 3D information visualization of the same terms. Both interfaces were used by volunteer students from grades five and six of an elementary school to locate terms within the taxonomy. The interfaces were evaluated according to whether the task was successfully completed, and if so, how quickly. The students' affective reactions to both interfaces were also collected through a questionnaire. Neither interface performed significantly better than the other in terms of task completion or task time; a majority of students found the conventional interface easier to use but the 3D interface more fun.
In an earlier article, we reported on an experimental evaluation of two interfaces designed to display a taxonomy of terms intended for use by children and relating to Canadian history (Large et al, 2009a). One interface used a conventional method for displaying the taxonomy through branching hierarchical levels, while the second interface utilized a two-dimensional visualization model based upon radial layout. Thirteen volunteer grade-six students from an elementary school in Montreal were asked to locate four terms within the taxonomy using both interfaces. Two usability metrics – successful completion of a task, and the length of time required to achieve this – were measured, and a number of affective factors were observed for each interface. Overall, neither interface demonstrated a clear superiority over the other.
One of the problems encountered with the two-dimensional radial layout was the labeling of those branches with many nodes. Our solution to this problem was to use a dynamic labeling scheme where only some of the leaf nodes were labeled at any given time, the criterion being the proximity of the cursor to the node being labeled. However, children encountered problems when they moved the cursor too quickly around the interface and thereby failed to identify terms as they flashed on and off the interface. We therefore decided to experiment with a three dimensional interface which seeks to overcome this perceived drawback in the two-dimensional representation.
We postulate that a 3D visualization model may help us to overcome some of the challenges of representing hundreds of nodes with their labels in a two-dimensional space. We report here on a comparative evaluation of two interfaces: the conventional display of the taxonomy used in the earlier experiment, and a new prototype 3D model.
Children face challenges ranging from selecting and correctly spelling appropriate keywords to coping with synonyms and homonyms when searching for information on the Web (Bilal, 1999, 2000, 2001, 2002a; Large & Beheshti, 2000; Large, Beheshti & Moukdad, 1999; Schachter, Chung & Dorr, 1998; Watson, 1998). Browsing subject taxonomies offers an alternative to keyword searching for children (Large et al, 2006). A taxonomy consists of hierarchically arranged categories (nodes) that progress from general to specific, and where each node is a subset of the higher level node (Blackburn, 2006). A menu of top-level terms is displayed; clicking on a term displays a second level, and so on, until an appropriate term with which to represent the information need is found. Such a taxonomy allows children to recognize concepts that are relevant to their information needs, potentially involving less cognitive effort than retrieving from memory terms to be used for a keyword search. However, taxonomies in practice may impose other obstacles because the child must be able to identify the appropriate term within the taxonomy's hierarchical structure; the greater the depth of the taxonomy, the greater the potential challenge. One solution could be to increase the breadth of the taxonomy and reduce its depth, resulting in longer lists of terms to be browsed at any particular level (Hutchinson, Druin & Bederson, 2007). An alternative solution is to use visualization techniques as a way to display more of the taxonomic structure on any one screen. Desclefs and Soto (1999) emphasize that information visualization can provide users with an overview of a hierarchy and enable them to visualize a specific node within the overall context, thereby facilitating navigation. The information visualization literature has provided a number of strategies to represent tree-shaped data, each with their own set of advantages and drawbacks. Some of the visualization paradigms include the hyperbolic tree layout (Lamping et al., 1995), the circular layout (Melançon and Herman, 1998), circle packing (Wang et al., 2006), and the radial layout technique (Eades, 1992) used in our previous study. An evaluation of some of the basic layouts, with a particular emphasis on space trade-offs, is given in Nguyen and Huang (2002), who concluded that each technique has limitations such as using the rectangular space of the screen to present a circular display, or poor performance on non-balanced trees.
A relatively popular visualization technique for displaying hierarchical information is the cone tree structure (see for example Carriere & Kazman, 1995; Cockburn & Mckenzie, 2000). First discussed by Robertson et al (1991), its goal is to maximize the effective use of screen space whilst minimizing user's cognitive load by visualizing the whole structure (Trinidad et al, 2008). A cone tree may be used to represent a hierarchy in a 3D space, where the ‘children’ are arranged around the circular base of the cone, with a node located at the apex of the cone. Using animation, the base of the cone may be rotated to view the selected node. Cone trees present an alternative for visualizing the taxonomy in three dimensions, as demonstrated by various applications such as Cat-a-Cone (Hearst and Karadi, 1997) and Lyberworld (Hemmje et al, 1994). The Cone also incorporates the findings of our previous research on the design of aesthetically pleasing interfaces for children, and includes balance, equilibrium, symmetry, unity, rhythm, and economy (Large et al., 2009b).
We constructed the taxonomy used in both interfaces for History Trek (http://www.historytrek.ca), a portal on Canadian history designed by an intergenerational team for children (Large, Beheshti, Nesset & Bowler, 2004; Large, Nesset, Beheshti & Bowler, 2006). It comprises 1397 terms hierarchically arranged to a depth of four levels under eight top-level topics: Places, People, Aboriginal Peoples, Everyday Life, Science and Technology, Transport, Government, and War (Table 1). The terms were derived from a subject analysis by the researchers of around 1000 web sites retrieved after searching on concepts taken from the grade-six curricula in Quebec and Ontario as well as an examination of print-based information resources designed for student use in elementary grades (Bowler, Nesset, Large & Beheshti, 2004).
Table 1. Distribution of subjects at different levels of the taxonomy
Science and Technology
Two interfaces were developed: a List interface demonstrating a conventional hierarchical list of terms, and a Cone interface representing the taxonomy in three dimensions. In the List interface, navigation up and down the four levels of the taxonomy is accomplished by clicking on individual terms (Figure 1). This design is a modification of the Subject Topics structure included in History Trek.
Cone, is a product of the current research project and is based on previous research (see Background). Navigation and term retrieval in Cone is accomplished by using the mouse and clicking on terms. The entire taxonomy is presented on the screen; zooming in and out allows the user to see the individual terms (Figure 2). The mouse is used to rotate the cone and to move up and down the hierarchy.
The Cone interface was implemented using Truevision 3D, a commercial, off-the-shelf software designed for constructing 3D environments and games.
The 14 students – ten boys and four girls11 – who participated in this study were volunteers from both grade-six and grade five classes in a Montreal, English-language elementary school. They were recruited after hearing a brief presentation about the research and its objectives by a researcher who visited their classes.
The task for each student was to locate four individual terms embedded within the taxonomy: two terms were from level three and two from level four. The four individual terms – Canoes, Vaccines, Eastern Townships and Mike Myers – were chosen for this test because they were likely to be familiar to any Montreal grade-six or grade-five student undertaking the test. Each term was printed on a plain card, along with a brief description:
Canoes – a way to travel by water
Vaccines – also called “shots”, a way to prevent you from catching diseases
Eastern Townships – about one hour's drive from Montreal
Mike Myers – a Canadian film star
The terms were located in the taxonomy as follows:
L1: Transport – L2: Boats – L3: Canoes
L1: Science and Technology – L2: Medicine – L3: Vaccines
L1: Places – L2: Provinces and Territories – L3: Quebec – L4: Eastern Townships
L1: People – L2: Jobs and Occupations – L3: Actors – L4: Mike Myers
The procedure was pre-tested with graduate students substituting for the elementary school students. Detailed instructions for the five research assistants on the project were prepared and they were given a short training session to ensure that the procedure would be executed smoothly and consistently. The procedure and the tasks were identical to those used in our previous study involving the two-dimensional radial model, with one exception. This time the students at the outset were given a task to familiarize them with the 3D environment. The Cone interface was likely to be significantly different from any used previously by the students, and a brief practice session would alleviate any misunderstandings or anxieties that might be caused by this new environment. The evaluation took place in May 2010 in the elementary school's computer lab after the regularly scheduled classes were finished. On the first day, ten students were individually tested, and in the subsequent two days the remaining four students were individually tested.
First, the purpose of the evaluation was briefly explained by a researcher who followed a script to ensure consistency. It was emphasized that the interfaces and not the student were being evaluated. Each student then answered a pre-questionnaire to collect information about the student's background and demographics. This was followed by the practice task. The student was asked to locate the term “Canada Day” in each interface to familiarize him/her with both the interfaces. The student was then asked to locate each of the four task terms in the taxonomy, two using the Lists version of the interface and two the Cone version. The research assistant noted on a checklist the student's affective behaviour, including observations on the student's level of understanding, excitement, and focus, as well as any obvious problems encountered or questions raised by the student. The tasks and the interfaces were randomized to counteract learning effects, i.e., half of the students used the List interface first, while the other half used the Cone first, and the terms were randomized to ensure that the terms were sought equally on both interfaces. All the conversations during the practice and test sessions were recorded. Finally, each student answered a post-questionnaire to gather affective reactions to the two interfaces.
To avoid student frustration, it was decided that the duration of any task search would be limited to a maximum of five minutes, after which the student would be shown how to find the term and the task would be judged as incomplete. In the event, that a student expressed his/her frustration over the lack of progress in a task search, the research assistant would intervene or terminate the session, and the task again would be recorded as incomplete.
The 14 students each completed four tasks, for a total of 56 tasks; 28 were undertaken on the List interface and 28 on the Cone interface. The start time was noted when the student began browsing the List/Cone. The end time was noted at the moment when the screen displayed “Bingo, you found the word”, or when the student did not wish to continue the task (incomplete task).
All 14 students were fluent English speakers, but four spoke languages other than English at home. The youngest was 10 years' old, two were aged 11 years, and the remainder were 12. They used the computer and the Web at least several times per week, and 12 used it daily. Their search engine of choice in all cases but one was Google. Only two students said that they never played video games, and the remainder were able to provide the names of their favourite games. The frequency of video game usage, however, varied from daily to less than once per month.
Of the 56 tasks assigned to the 14 students, 10 were unsuccessful (Table 2). All the students successfully completed the task “Canoes” on both interfaces. These results are similar to our previous study, where out of 52 searches, 11 were incomplete, but all completed the “Canoes” task. Six tasks for the List and four tasks for the Cone were not completed. The numbers in Table 2 shows the actual times, after which the students did not wish to continue with the task. In one case, the research assistant intervened after five minutes to stop the task. The other students resigned from their tasks well before the allotted time limit of five minutes.
Table 2. Incomplete Tasks
The overall average time for 47 successfully completed tasks is 94.9 seconds (standard deviation =86.05, median=75.0). Table 3 shows the mean times for the four tasks for each interface. While the mean time for the tasks on the List interface is less than the times recorded for the Cone interface, the T test (t=1.145, df=45, p=0.258) shows no statistically significant difference between the mean times for the two interfaces.
Table 3. Mean times (seconds) for each interface
A one-way analysis of variance indicates a significant difference in the overall mean times among the tasks (F3,43=6.660, p=0.001). The combined data for both interfaces show that the level-four tasks required more time than the level-three tasks. The “Eastern Townships” task took the longest amount of time, requiring more than three minutes on average versus the overall fasters task, “Canoes”, at less than 50 seconds (Table 4).
Table 4. Overall Mean times (seconds) for different tasks
Table 5 shows the average times for each task for each interface. For all the tasks the average times are higher for Cone than List. However, the results of the T-test and the Median Test show that none of these differences are statistically significant. These data may indicate a trend, but small sample sizes with large standard deviants (Figures 3 and 4) do not allow us to conduct more in-depth statistical analysis, and therefore any generalization has to be speculative.
Table 5. Mean times (seconds) by task
Med Test p=
The post-questionnaire probed the students' subjective reactions to the two interfaces. They were asked to rate each interface on a Likert scale (1 low to 5 high) in terms of how they liked the way in which words (terms) were displayed: the mean score for List was 3.79 and for Cone 3.50, while the median and the mode score for both interfaces were 4.0. For both interfaces, only two students awarded a maximum score of five points.
There was a significant negative correlation between student preference for display of words in the Cone interface and their video game usage (Spearman's rho= −0.733, p=0.003). The more frequent users of such games were less likely to prefer the display of words in the 3D visual interface.
The students were also asked which interface they preferred when judged by four criteria:
ease of use
fun to use
Although a majority found List easier to use than Cone, the latter was rated more fun (Table 6). A majority also considered that it was faster to locate a term using List than Cone, an estimate confirmed (but not significantly) by the actual recorded times (see Table 2). A slight majority expressed their preference for the List over the Cone.
Table 6. Affective student reactions (number of students)
From their comments to the research assistants and in the post-questionnaire responses, it is clear that almost all the students grasped at the outset what they had to do. Students were fairly articulate in their preferences, and likes and dislikes. One of the younger boys said the Cone interface was “cool, as if you're travelling through history.” He then elaborated that the Cone “is faster; it takes a little time getting used to it, but when you do, it is faster to use.” Another older boy, comparing the List to the Cone interface, said that the latter was “easier but [I] don't know how to turn around [rotate the view]”. He went on to elaborate, stating that in the Cone interface “you can see all the words you need to see on screen”, and it is more fun to use because “you need to look around a lot”. One student, who favoured the List interface, stated “I like how words are organized in the List. This [Cone] is a little confusing for me.” However, she admitted that “the Cone is prettier and fun, it has colors and it's pretty to look at, and it's animated.” Another student preferring the List, said “they [both interfaces] have the same topics, but the Cone was much harder. The List narrowed it [the topics] down a lot more.” He pointed out his frustration and problems with navigation, stating that “the Cone was annoying to use because I messed up sometimes, I had to zoom out all the way and click at the top [the apex], not like the List where I can click on ‘people’ [top level term in the taxonomy] and go back if I make a mistake.”
One research assistant's observation revealed that the student, who completed only one of the four tasks (Table 1, Student 12), had trouble understanding the hierarchical structure of the taxonomy, regardless of which interface he used. He would correctly choose the appropriate terms at the top level of the taxonomy, but would not explore further the lower branches. Nevertheless, for him the Cone was the clear choice: “I get to see all my options, but the other one [List] you have to scroll down.” Another student, who completed two out of the four tasks (Table 1, Student 8), seemed to be interested in the experiment, but was easily distracted. The research assistant had to intervene several times during the session to ensure that the student focused on the task.
Overall reactions to the interfaces were varied. List was seen to be more ‘organized’ and easier to use, whilst Cone's animation and colours were viewed to be more fun for searching. Several students found List more familiar, and said that Cone had to be explored initially to understand the navigation and movements through different levels of the taxonomy.
Discussion and Conclusions
The experiment reported here compares a conventional interface with a 3D visual interface for displaying a taxonomy intended for children. The experiment involved 14 student volunteers from grade five and six elementary classes in an English-language, Montreal school. Five research assistants and one researcher conducted the experiment in the school's computer lab. None of the participants had previously encountered the web portal, History Trek, and therefore were unfamiliar with the specific taxonomy employed in its Topic Search (subject taxonomy). Nor had any of them encountered an information visualization interface similar to the Cone. Effectively, they approached both the conventional List and the 3D Cone interfaces with little or no experience of such designs.
While the students were interested in the experiment and keen to find the terms, they did not always find it straightforward to locate the relevant top-level entry point in the taxonomy. The task “Eastern Townships” proved particularly difficult for them, as a few searched under other top-level headings than the correct one – “Places”. Several students seemed unsure about the taxonomy structure or the terminology, regardless of the interface they used, and in several cases quickly became frustrated and never actually located the term.
The students undertook 56 tasks, of which 10 (18%) were unsuccessful. Such failures may be due to the interfaces: 6 out of 10 failures were on the List interface, while the remaining 4 were on the Cone interface. However, a more likely explanation for uncompleted tasks is the level of the term within the taxonomy. The two terms at level 4 (rather than level 3) within the hierarchy, perhaps not surprisingly, accounted for 7 of the 10 (70%) failures. It is interesting to note that failures did not decrease as students gained more practice with the two interfaces or became more familiar with the underlying concept of a taxonomy.
Although overall the tasks were completed slightly faster on the conventional List interface than the Cone interface, the difference is not statistically significant. As regards the individual tasks, no statistical significant differences were found between the List and the Cone. While the average times for completion of each task were higher for the Cone (particularly in the case of ‘Mike Myers’), the standard deviations or the variance among students were too large to yield any significant findings (see Figure 4). We can only speculate about these variations; they may have been a result of different cognitive styles among students. A much larger sample size may reduce the variance and produce more robust result, or afford us other variables to explain the discrepancies in task completion times between students.
One interesting finding is that those students who played video games frequently seem to dislike the 3D visualization interface more than those who are infrequent or non-users. We found a similar negative correlation in our previous study on the 2D visualization, and speculated that two possible rationalizations could explain this finding. First, the gamers tend to move and click at faster speeds than less frequent players, which may result in failure to spot relevant terms that flash on and off the screen as the cursor travels on its way. A second possible explanation is that gamers had higher expectations of visual and navigational features that were not met in the non-gaming environment of the Cone.
The affective reactions of the students to the conventional and 3D visualization interfaces were also mixed. When asked to rate how they liked the way the terms were displayed in List and Cone, there was no significant advantage for either interface, although List scored slightly higher. A clear majority of the students said they found List easier to use, but Cone was considered more fun. A majority of students believed that List was faster than Cone, and the actual timings support their conjecture. Very similar results were also observed in our previous study. It is noteworthy that Cockburn and McKenzie (2000), when comparing two interfaces, a Cone tree and a conventional hierarchical tree, in an experiment involving adults, reported somewhat similar results. In that experiment, while the 12 adult participants were significantly slower in locating terms in the Cone interface, they were very “enthusiastic” about the 3D environment (p. 436).
This was a small-scale experimental study, with certain limitations that should be taken into account when evaluating its findings. The sample of students was fairly homogenous and although they did range in age from 10 to 12 years, it would be unwise to generalize the findings either to younger or older children. A larger heterogeneous sample of children may have provided us with more explanatory variables for certain results such as the negative correlation between gaming and views about the visual interfaces. With a larger sample, we may be able to abandon the “simple locate, count, and compare tasks” in favour of exploratory tasks for a deeper understanding and insight into information visualization (Andrews, 2006, p. 1).
This study is the second stage of a larger project which is investigating ways in which a taxonomy might be presented to children so as better to facilitate their rapid and accurate retrieval of relevant information. Overall, its results were inconclusive. Neither interface design outperformed the other significantly in task completion success, or time to task completion. The conventional interface design was judged easier to use, but the 3D visualization interface more fun to use. This does suggest that while visualization might not perform better under test conditions, it is more likely to engage young users in search tasks.
We are grateful for the participation in this study of the 14 students, and for the cooperation of their teachers and school. The research reported here is part of a larger study funded by the Social Sciences and Humanities Council of Canada. The authors are grateful for the assistance of Nouf Khashman, Svetlana Aksenova and Megan Stecyk.
Given the results of our previous research that gender does not play a significant role in the performance of children when searching for terms in a taxonomy (Large et al, 2009a), a gender-balance sample was not considered when recruiting volunteers for this study.