Department of Computer Science, University of Massachusetts at Boston, 100 Morrissey Boulevard, Boston, MA, 02125-3393. E-mail: email@example.com
Color coding is used to guide attention in computer displays for such critical tasks as baggage screening or air traffic control. It has been shown that a display object attracts more attention if its color is more similar to the color for which one is searching. However, what does similar precisely mean? Can we predict the amount of attention that a display color will receive during a search for a given target color? To tackle this question, two color-search experiments measuring the selectivity of saccadic eye movements and mapping out its underlying color space were conducted. A variety of mathematical models, predicting saccadic selectivity for given target and display colors, were devised and evaluated. The results suggest that applying a Gaussian function to a weighted Euclidean distance in a slightly modified HSI color space is the best predictor of saccadic selectivity in the chosen paradigm. Hue and intensity information by itself provides a basis for useful predictors, spanning a possibly spherical color space of saccadic selectivity. Although the current models cannot predict saccadic selectivity values for a wide variety of visual search tasks, they reveal some characteristics of color search that are of both theoretical and applied interest, such as for the design of human–computer interfaces.
Searching for a visually distinguished item is not only a routine task in everyday life but also essential to operating human–computer interfaces for critical functions such as air traffic control, baggage screening, and computer assisted surgery (e.g., Edwards et al., 2000; Remington, Johnston, Ruthruff, Romera, & Gold, 2000; Wolfe, Horowitz, & Kenner, 2005). To direct the operator's attention, many interface designers intuitively choose to employ color features. Clearly, if operators are searching for a particular color A, then a color similar to A will also attract their attention, whereas A's complementary color is less likely to be attended to. For the further investigation of this effect, similarity must be more explicitly defined. How can we predict for a given target color, A, how much attention a display color B will receive? Can we sketch out a specific color space of visual search that would allow us to make such predictions for any two colors A and B?
Empirical studies have indeed yielded evidence for an outstanding ability of color information to guide visual search. In a typical visual search experiment, the participant's task is to decide whether a display composed of various search items contains a previously specified target item (see Wolfe, 1998, for a review). Several visual search studies have investigated response times and error rates as functions of the display colors' positions in continuous color spaces (e.g., Bauer, Jolicoeur, & Cowan, 1996, 1999; D'Zmura, 1991). One of the most important findings relates to the CIE XYZ color space, using one target color and two distractor colors: When in the (x, y) plane of the color space the target color is located on a straight line connecting the two distractor colors, search is much more difficult than in all other cases (Bauer et al., 1996; D'Zmura, 1991). This result was later generalized for the entire (x, y, z) space and a greater number of display colors: Search difficulty was found to be greatest when the target color was within the convex hull of the other display colors (Bauer et al., 1999). Some researchers, however, doubt the entirely continuous nature of such effects and propose that broad categorization of color (e.g., “reddish” vs. “bluish” colors) plays an important role in their processing (Yokoi & Uchikawa, 2005).
A slightly different approach to the study of visual search is the analysis of participants' eye movements during search performance (e.g., Findlay, 2004; Pomplun, Shen, & Reingold, 2003; Scialfa & Joffe, 1998; Williams & Reingold, 2001). Although not all shifts of visual attention are accompanied by corresponding gaze shifts, eye movements were found to provide a valid measure for the display items attended to during search, if one accepts a certain level of noise in the data (see Findlay, 2004). If different subsets of display items share different features with the target, then the gaze information can be used to compute saccadic selectivity—that is, the proportion of saccadic endpoints aimed at each type of target feature in the display.
One general finding from these saccadic selectivity studies is that if the display items have, among other features, clearly distinguishable colors, then participants' eye movements tend to be guided by the target color. For example, Williams and Reingold (2001) used target items of particular color, shape, and orientation, and three groups of additional display items; each of them sharing exactly one of those three features with the target. By attributing each saccadic endpoint to its nearest display item, it was found that saccadic selectivity for the target color was about 70%, whereas it was only about 15% for shape and orientation.
Although the aforementioned line of research has pointed out the dominance of the color dimension in guiding visual attention during search, to date no study has systematically measured the effect of color similarity on the selectivity of saccadic endpoints. This work is a first step toward establishing a useful color space of saccadic selectivity in visual search tasks. In Experiment 1, a sample set of 64 colors was chosen. A simple color search task was used to measure saccadic selectivity for every combination of display and target colors within this set. Undoubtedly, this was a very ambitious plan: Even if we assume symmetry between display and target colors, there are still 2,016 selectivity values to be measured. To determine each individual value reliably, many hundreds of experimental sessions would have been necessary, which did not seem advisable for this first, explorative study. Therefore, we decided to tolerate substantial noise in the data and take advantage of the large number of 64 colors to obtain a first rough mapping of the color space of saccadic selectivity. After visualizing the data, a large number of quantitative models of saccadic selectivity were devised and evaluated. Experiment 2 was identical to Experiment 1 except for the set of colors it employed. The results of Experiment 2 were used to further evaluate the models of saccadic selectivity.
1. Experiment 1
This research was carried out with the assistance of 20 participants of ages 19 to 36. Participants were paid a $10 honorarium. Of these, 19 were students and 1 was a member of the faculty at the University of Massachusetts at Boston, and all of them had intact color vision.
Stimuli were presented on a 21-in. Dell P1130 monitor (CIE chromaticity values—red: x = 0.625, y = 0.340; green: x = 0.275, y = 0.605; blue: x = 0.150, y = 0.065; color temperature: 9,300 K). Under the experimental conditions, the black screen had a luminance of 2.3 cd/m2. The resolution of this monitor was set to 1,024 × 768 pixels and its refresh rate to 85 Hz. Participants sat approximately 60 cm from the screen. The horizontal and vertical viewing angles of the stimuli were approximately 34° and 26°, respectively. An SR Research EyeLink-II system was used to track eye movements. The average error of visual angle in this system is 0.5°, and its sampling frequency is 500 Hz. A handset or “game pad” was used to register the participants' manual responses.
A set of 64 different colors was used, which were composed of all possible combinations of four luminance levels of red (0.0, 2.0, 5.8, and 12.8 cd/m2), green (0.0, 7.1, 22.2, and 50.5 cd/m2), and blue (0.0, 1.4, 3.8, and 8.0 cd/m2). The four levels for each of these three constituent colors were chosen to be approximately perceptually equidistant (see Pinoli, 1997). Target displays, used to indicate the target color, filled the entire screen with one of these 64 colors. Although it is known that the size of an area slightly influences the perception of its color (Kutas, Gócza, Bodrogi, & Schanda, 2005), the entire screen was used for presenting the target color to present its slight variation across the cathode ray screen and avoid contrasting the target color with a particular background color. Search displays in which participants had to find the target color divided the screen into an array of 8 × 8 rectangles of equal size (4.3° horizontally and 3.2° vertically). Each of these rectangles showed a different color so that each color from the set of 64 appeared exactly once (see Fig. 1).
The experimenter began each experiment by providing the participant with task instructions, fitting the eye-tracking headset to the participant, and then calibrating the eye tracker. Participants started each trial and at the same time performed a drift correction of the headset by pressing a button while fixating on a central marker. Each trial began with the presentation of a target screen for 2 sec, during which the participants were to memorize the target color. Subsequently, a search display appeared. Participants were to search the array for the target color and, while fixating on this color, to press a button to terminate the trial. If a participant did not press the button within 4 sec after search-display onset, the trial would “time out” and terminate. After the termination of every trial, a black and white frame was shown around the target color to provide feedback to the participants about their accuracy. Each of the 64 colors served as the target in four randomly chosen trials, resulting in a total of 256 trials per participant. The positions of the 64 colors in each display were randomized and counterbalanced across trials.
1.2. Results and discussion
The average trial duration was 2,319 msec, and the mean duration of fixations was 207 msec. In 32.5% of all trials, at least one of the last two fixations was located inside the target rectangle. Fig. 1 shows a sample scanpath generated by one of the participants. Fixations from all trials were included in the analysis of saccadic selectivity if they started after the onset of the search display, ended before the manual response or timeout, and were not located on a rectangle of the target color. For each target color T, we computed the average distribution of saccadic endpoints across the 64 display colors, using a scale from 0 to 100. A display color D would receive a value of 0 if no saccadic endpoints landed on a rectangle of color D, given target color T; and 100 would mean that for target T all saccadic endpoints from all participants landed on color D. Consequently, the selectivity data formed a 64 × 64 matrix with zeros on its diagonal because we excluded fixations on the target color. This saccadic selectivity matrix was roughly symmetrical, which is in line with the premise that saccadic selectivity for a display color D increases with greater similarity—a symmetrical concept—between D and the target color T. Therefore, we computed the arithmetic mean of all symmetrical pairs in the matrix, which resulted in 2,016 values indicating the mutual saccadic selectivity between any two colors from the chosen set.
To get a rough sketch of the color space of saccadic selectivity (i.e., a representation in which proximity of two colors indicates their mutual saccadic selectivity), we employed the technique of multidimensional scaling (e.g., Cox & Cox, 2001). This technique maps a similarity or distance matrix for a set of objects onto a multidimensional (usually 2-dimensional or 3-dimensional) abstract space in which the objects are placed in such a way that more similar ones are separated by a smaller Euclidean distance. Fig. 2 shows a stereoimage pair visualizing the result of 3-dimensional multidimensional scaling of the present selectivity data (PROXSCAL algorithm, spline transformation degree 3, one interior knot, simplex start configuration, 100 iterations, resulting normalized raw stress 0.11). By crossing one's visual axes to fuse the two images into one, it can be seen that the 64 colors are roughly placed on the surface of a sphere. Notice that this spherical shape is not a consequence of the specific algorithm used, but solely reflects the pattern of mutual saccadic selectivity between colors.
In Fig. 2, hues fall about the approximately vertical axis much as they do in the HSI color space. It also seems that along this axis, from the bottom to the top of the sphere, the intensity of colors tends to increase. Saturation, however, does not seem to play an important part, as there are no clusters of high- or low-saturation colors. Probably the most conspicuous exception from this rough picture is the color white, which is not located at the top of the sphere but close to the group of light blue colors. It is likely that the high color temperature of the monitor (9,300 K) contributed to this result. Regarding the hypothesis of color categorization (Yokoi & Uchikawa, 2005), Fig. 2 suggests a continuum of colors rather than strict categories, which would be indicated by tight clusters around basic colors (Berlin & Kay, 1969). Although the spatial distribution of colors on the sphere is not exactly homogeneous, there are transitions such as from the purple colors toward bluish ones (leftward) or reddish ones (rightward). Although categorization, possibly through verbal memorization of colors, cannot be ruled out, it does not seem to be the predominant factor determining saccadic selectivity.
What is the most precise and useful mathematical description of this color space? To have a baseline for evaluating a variety of mathematical models, we first devised an overly simple model (Constant model), which assumes that color has no effect on saccadic selectivity at all. In other words, this model maintains that all display colors receive the same amount of saccadic endpoints, regardless of the target color. For the actual modeling, our aim was to find functions with concise mathematical descriptions that estimate the mutual saccadic selectivity of two given colors as accurately as possible. As first approaches, we modeled the mutual saccadic selectivity m between two colors c1 and c2 as linearly decreasing with the colors' weighted Euclidean distance in the four standard color spaces RGB, HSI, CIE XYZ, and CIE Lab (see Table 1, rows 2 to 5). For each color space, the free model parameters were numerically determined to minimize the mean square error (MSE) between the computed and the empirical saccadic selectivity across all 2,016 color pairings. In particular, for the HSI space, we evaluated three different ways of computing the hue variable for a given color: its polar angle in CIE XYZ relative to the point (1/3, 1/3), its polar angle in CIE Lab relative to the point (0, 0), and its standard definition (derived from the RGB space). For these and all following HSI-based models, the CIE XYZ version of hue computation was found to be the most accurate. Thus, throughout the remainder of this text, this version is used.
Table 1. Equations with fitted parameters and resulting mean square errors for some of the evaluated models. Notice that some of these variables, such as the hue variable in the HSI space, correspond to angles. These angles are given in radians, and differences between them are measured in such a way that they never exceed π. The values of R, G, and B range from 0 to 1
Although the HSI model produced the smallest MSE, it did not differ significantly from the other three linear approaches: all ts(2,015) < 2.52, ps > .10. All of these linear models, however, outperformed the Constant model: all ts(2,015) > 8.46, ps < .001. Notice that, due to the low number of data points per cell, only the cumulative saccadic selectivity for all participants was analyzed, and the standard error was computed across the 2,016 color pairings. All results from t tests were Bonferroni adjusted.
To devise models with better fit to the empirical data, we tested a large number of linear, logarithmic, polynomial, and exponential functions and their combinations in all of the four color spaces. For all of these functions, computations in the HSI space either outperformed those in the other three color spaces or were statistically identical to them. Moreover, it was found that Gaussian models, which simply apply a Gaussian function to the weighted Euclidean distance, provided better fits than all other approaches. These models are of the form shown in Table 1 for the HSI Gauss model.
Accordingly, the HSI Gauss model achieved the best fit in this competition. Fig. 3 shows the significant improvement of the HSI Gauss model over the linear HSI model: t (2,015) = 5.89, p < .001. Because the difference between two colors ranges from 0 to π in their hue, but only from 0 to 1 in their saturation and intensity, the fitted parameters shown in Table 1 suggest that hue is dominant in guiding attention, followed by intensity, whereas saturation is much less important. This finding is in line with the results of the multidimensional scaling shown in Fig. 2.
The aforementioned evidence for the small impact of the saturation dimension on saccadic selectivity raises the question whether saturation can be completely disregarded without losing significant predictive accuracy. To answer this question, we implemented and evaluated the HI Gauss model as defined in Table 1. Fig. 3 shows that the HI Gauss model is only slightly, but significantly, less accurate than the HSI Gauss model that accounts for all three dimensions, t (2,015) = 3.62, p < .01; and outperforms the linear HSI model, t (2,015) = 5.54, p < .001. Finally, motivated by the results of the multidimensional scaling, we implemented the Sphere model, which computes saccadic selectivity based on the Euclidean distance between colors arranged on a spherical surface according to their hue and intensity. The most accurate approach for this model is given in the bottom row of Table 1, where the variable d—set to 1.68 in the fitted model—determines whether brightness ranges from pole to pole (d = 2) or is restricted to a maximum distance of d/2 from the equator (0 ≤ d < 2). As shown in Fig. 3, the MSE of the Sphere model was slightly greater than the one for the HSI Gauss model, t (2,015) = 3.24, p < .05; and minimally, but not significantly, smaller in comparison to the HI Gauss model, t (2,015) = 1.71, p > .50.
In summary, the HSI Gauss model yields the best fit among the three-dimensional models, whereas the Sphere and HI Gauss models are the best fitting two-dimensional ones. To test whether this result generalizes for different color sets than the one used in Experiment 1, we conducted Experiment 2, which employed the same experimental paradigm as Experiment 1 but a different set of colors.
2. Experiment 2
Another 20 participants (18 students, 1 faculty member, and 1 staff member of the University of Massachusetts at Boston, aged 19 to 42) took part in Experiment 2. All of them had intact color vision and were paid a $10 honorarium.
2.1.2. Apparatus, materials, and procedure
Experiment 2 was identical to Experiment 1 except for the choice of the 64 colors, which were selected along the dimensions of the HSI color space. The 64 colors resulted from all possible combinations of eight levels of hue (0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°), two levels of saturation (0.2 and 0.8), and four levels of intensity (0.4, 0.6, 0.8, and 1). The unequal numbers of levels were chosen to reflect the differences in selectivity across dimensions as suggested by the results of Experiment 1.
2.2. Results and discussion
The average trial duration was 2,266 msec, and the mean duration of fixations was 209 msec. In 31.9% of all trials, at least one of the last two fixations was located inside the target rectangle. None of these values differed significantly from those in Experiment 1: all ts < 1.
The same models that were fitted to the saccadic selectivity data in Experiment 1 were evaluated with the data obtained in Experiment 2, without fitting their parameters to the new data set. Fig. 3 shows the results for the same six models that were displayed for Experiment 1. The pattern of results in Experiment 2 was very similar to Experiment 1, except that the variance of the selectivity data was higher in Experiment 2. Like in Experiment 1, the Constant model generated a greater error than all linear models: all ts(2,015) > 2.34, ps < .05. Among the linear models, the HSI model performed significantly better than all of its competitors, all ts(2,015) > 7.33, ps < .005; but produced a greater error than did the HSI Gauss model: t (2,015) = 5.19, p < .001. Again, there was no statistical difference in MSE between the HI Gauss and Sphere models: t (2,015) < 1. Although they outperformed the HSI model, both ts(2,015) > 4.21, ps < .001; their error was greater than the one for the HSI Gauss model: both ts(2,015) > 5.47, ps < .001.
By comparing the ranges of error values between Experiments 1 and 2, it can be seen that in each experiment the error of the best performing model—the HSI Gauss model—was about 50% below the error produced by the Constant model, which served as an error baseline. Moreover, the relative performance of the different models, whose parameters were fit to the data of Experiment 1, remained approximately the same in Experiment 2. It thus seems justified to say that these models illustrate some properties of the color space of saccadic selectivity that apply beyond the particular set of colors chosen in Experiment 1.
The visualization and mathematical modeling of saccadic selectivity for color provided a rough map of some of the characteristics of the underlying color space. Among all models, the HSI Gauss model achieved the best fit to the data in Experiment 1 and demonstrated the greatest accuracy at predicting selectivity in Experiment 2. This model applies a Gaussian function to the weighted Euclidean distance of two colors in a slightly modified HSI space in which the hue is derived from the CIE XYZ color space. Simplifying this model by disregarding the saturation dimension led to the HI Gauss model, with only slightly reduced accuracy. The Sphere model, which was inspired by the results of the data visualization, also uses only the hue and intensity dimensions, which form a spherical color surface in a three-dimensional abstract space. Although the MSE generated by this model was slightly lower than that for the HI Gauss approach, these data did not allow a statistical distinction between the two models.
The relatively poor fit of the models based on the CIE Lab color space is surprising, as this space was specifically designed to approximately linearize perceptual color differences (see Wyszecki & Stiles, 2000). However, these perceptual differences were measured foveally, whereas saccadic selectivity during visual search also depends on peripheral color discriminability. Sensitivity to green color, for instance, decreases more rapidly with greater retinal eccentricity (see Newton & Eskew, 2003), which is in line with the area of green hues being less focused in the modified HSI color space than in the CIE Lab space. Future studies need to quantify the possible contributions of perceptual color memory, peripheral color discriminability, and color contrast on saccadic selectivity.
We have to be cautious when interpreting the current results because of the substantial noise introduced by the huge number of dependent variables and by the possibility of covert shifts of attention. Moreover, changes in the display such as modifying the size or shape of the color regions, changing the proportion of individual colors, or introducing other stimulus dimensions would modify saccadic selectivity in ways that could not be predicted by the current models. This study focused on saccadic selectivity for color by itself, in an abstract and isolated form. Before these results can be applied to real-world problems such as the design of efficient human–computer interfaces, further research is needed to refine the results and integrate them with broader models of visual attention.
This work was supported by an Undergraduate Research Award to Emily C. Higgins by the University of Massachusetts at Boston.
We thank Wayne Gray, Alex Hwang, Ruth Rosenholtz, and an anonymous reviewer for their helpful comments on an earlier version of this text.