A Dynamic Context Model of Interactive Behavior

Authors


should be sent to Wai-Tat Fu, Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801. E-mail: wfu@illinois.edu

Abstract

A dynamic context model of interactive behavior was developed to explain results from two experiments that tested the effects of interaction costs on encoding strategies, cognitive representations, and response selection processes in a decision-making and a judgment task. The model assumes that the dynamic context defined by the mixes of internal and external representations and processes are sensitive to the interaction cost imposed by the task environment. The model predicts that changes in the dynamic context may lead to systematic biases in cognitive representations and processes that eventually influence decision-making and judgment outcomes. Consistent with the predictions by the model, results from the experiments showed that as interaction costs increased, encoding strategies and cognitive representations shifted from perception-based to memory-based. Memory-based comparisons of the stimuli enhanced the similarity and dominance effects, and led to stronger systematic biases in response outcomes in a choice task. However, in a judgment task, memory-based representations enhanced only the dominance effects. Results suggested that systematic response biases in the dominance context were caused by biases in the cognitive representations of the stimuli, but response biases in the similarity context were caused by biases in the comparison process induced by the choice task. Results suggest that changes in interaction costs not only change whether information was assessed from the external world or from memory but also introduce systematic biases in the cognitive representation of the information, which act as biased inputs to the subsequent decision-making and judgment processes. Results are consistent with the idea of interactive cognition, which proposes that representations and processes are contingent on the dynamic context defined by the information flow between the external task environment and internal cognition.

A growing trend in cognitive science is the investigation of the hypothesis that the human cognitive system is coupled to the perceptual-motor system and to the external world (Carlson, 1997; Clark, 1989; Clark & Chalmers, 1998; Gray, Sims, Fu, & Schoelles, 2006; Wilson, 2002). In contrast to the traditional view that the cognitive system is central and the perceptual and motor systems are peripheral input and output devices, proponents of embodied or interactive cognition claim that cognition should be understood in the context of its relationship to a physical body that interacts with the external world. In particular, because humans are believed to be well adapted to continuously processing information from the outside world, proponents of interactive cognition argue that the unit of analysis of the mind should contain both the cognitive and perceptual-motor operations, and that the interaction with the external environment should play a critical role in the analysis of the integrated cognitive systems.

One argument for interactive cognition is the general tendency to offload complex cognitive computations to the external environment to overcome our limits in information-processing ability, exemplified by our reliance on post-it notes, smartphones, or other tools to act as parts of our “extended mind.” Given that information in the external environment is often more stable and available than our internal memory, we only need to harvest them right before it is needed to reduce working memory load. Empirical studies also show that people tend to rely much on assessing information in the external environment to minimize use of internal memory. For example, in the Blocks-World task by Ballard, Hayhoe, Pook, and Rao (1997), participants were instructed to reproduce patterns of colored blocks by dragging them on a computer screen into a work area. Eye-movement data showed that participants tended to strategically fixate on the color and location information of each color block right before they are needed. Ballard et al. (1997) argued that participants adopted a minimal memory strategy that exploited the external world as an information source to minimize encoding of information in working memory. Similarly, in the study by Kirsh and Maglio (1994), expert Tetris players were found to make rapid rotations of blocks to decide where to drop them on the screen to offload mental computations to rapid perceptual-motor operations that directly manipulate information in the external world. Other studies also suggest that offloading computations to the external environment seem to be a natural response to overcome our limited cognitive capabilities.

Recent studies, however, have shown that the interactions between external and internal representations may be more complex and dynamic. For example, O’Hara and Payne (1998) showed that when the interaction costs imposed by the computer interface was increased by introducing a 2-s lockout time (e.g., when a user is waiting for a system response), participants relied more on their working memory to compute a solution in the eight-puzzle problem. Participants were found to perform more mental planning, which led to better overall learning of the task structures and chunking of action sequences. These results suggest that (a) the minimal memory strategy may not be always preferred, and (b) switching between processing information in the external world and in working memory may lead to changes in subsequent cognitive representations and processes that act on the information.

Consistent with these findings, the theory of soft constraints (Gray et al., 2006) proposed that the control process behind perceptual- and memory-based operations should be characterized by a sequence of moment-to-moment cost–benefit tradeoffs afforded by the structures of the interactive environment (Ballard et al., 1997; Carlson, 1997; Fu & Gray, 2006; Fu & Pirolli, 2007; Gray & Fu, 2004Kirsh & Maglio, 1994). In contrast to theories that assume minimal memory use, the theory of soft constraints predicts that when the cost of accessing information in the external world is high, people tend to rely more on working memory to store information. In addition, the theory predicts that as the costs of interaction increase, people will progressively increase the encoding of information into working memory, such that the precise mix of perceptual-motor and cognitive operations can be predicted by the interaction cost imposed by the environment and the benefit of the acquired information (Fu, 2007; Fu & Gray, 2006; Fu & Pirolli, 2007; Gray & Fu, 2004; Gray et al., 2006). Assuming that the control system is dependent solely on the cost–benefit tradeoffs between operations, the theory of soft constraints predicts that the decision on where to access information is indifferent to the information source (i.e., either external environment or internal memory). In other words, the external environment, similar to other internal representations, is an extension of the cognitive system that controls when to offload cognitive operations onto the environment based on the relative assessing costs of information from the environment.

1. Dynamic context effects in interactive environment

Although the theory of soft constraints is successful in explaining when people may choose to assess information-in-the-world rather than information-in-the-head, what is still unclear is whether the tradeoff may also influence subsequent cognitive representations and processes that take information from different sources as inputs, and how they may lead to predictable biases in response outcomes. Indeed, given that representations derived from perceptual-motor and cognitive operations are often different, one may expect that differences in representational inputs may induce systematic biases in subsequent processes and eventually influence response outcomes. In other words, differences in the characteristics of the task environment not only may influence how information will be assessed but also how they will be represented and processed in the task (Fu & Dong, 2010; Fu, Kannampalli, Kang, & He, 2010).

There has been a vast amount of research showing that different representations may lead to systematic differences in subsequent decision and judgment outcomes. For example, research has shown that although perceptual judgment on relative line lengths seems accurate and unbiased (Cleveland & McGill, 1985; Hochberg, 1978), when the same stimuli are encoded in memory, values of the stimuli recalled from memory are often systematically biased by top–down processes, which are influenced by previous experiences with similar stimuli (Huttenlocher, Hedges, & Duncan, 1991; Huttenlocher, Hedges, & Vevea, 2000; Poulton, 1985; Tversky & Schiano, 1989). Huttenlocher et al. (1991, 2000) showed that memory encodings of perceptual stimuli were influenced by their categorical coding: Two stimuli that are actually equidistant may be judged as more similar if they are from a common category than if they are from different categories; and a stimulus may be judged closer to the center of the category that it belongs to than it actually is. Results from these studies suggest that although the control of information accesses is indifferent to the sources, higher-level representations and processes may depend on whether information is coming from the external world or from internal memory. In other words, the dynamic context, defined by the mix of internal and external representations of stimuli, may influence the subsequent representations and processes when external objects are interpreted and judged.

Researchers of cognitive science have developed models to characterize how inputs from the external stimuli per se and from internally encoded knowledge are incorporated during the interpretation, recognition, or judgment of the stimuli (Huttenlocher, Hedges, Lourenco, Crawford, & Corrigan, 2007; Ullman, 1984). There seems to be agreement among researchers that while the bottom–up perceptual processes can provide the fine-grained uninterpreted representation of stimuli, more complete representations of stimuli depend on top–down processes. One common way to demonstrate the top–down influence is to compare the interpretation or judgment outcomes of the same set of stimuli in different contexts. The idea is that if the same stimulus is interpreted or judged differently in different contexts, then the difference is likely the consequence of the differing top–down processes induced by the contexts. For example, Tversky and Schiano (1989) asked participants to remember the angle that a straight line made with the horizontal axis. One group of participants was told that the line represented the relationship between two variables on a graph (the graph condition); another group of participants was told that the line represented a bike path on a map (the map condition). Results showed that participants in the graph condition had distorted memories of slopes that were biased toward this 45° imaginary line, but the same distortion was not observed in the map condition. Tversky and Schiano (1989) explained the results by assuming that people tended to use the imaginary 45° line as a natural anchor to interpret and encode, for example, whether the relationship between two variables indicates “rapid growth” or “slow growth.” Thus, the recalled slopes were systematically biased to be closer to this anchor. Because in the map condition participants were given a different context that did not encourage the use of the same anchor, no systematic bias was observed. In other words, the context of a graph induced a top–down representation of the stimuli (based on the imaginary 45° line) that was different from that in the map condition.

The perceptual judgment model by Huttenlocher et al. (1991, 2007) provides details about how inputs from bottom–up and top–down processes are combined in perceptual judgment. Specifically, they showed that while the bottom–up process provides fine-grain inexact (i.e., noisy) information about the stimulus, category information of the stimulus provides a course-grain estimate of the dimensional values of the stimulus. When recalling a particular dimensional value of the stimulus from memory (e.g., the length of a line), the category information will act as an anchor that combines with the fine-grain value during recall. The model shows that the way the categories are formed and the process that adjust the final estimate based on the category information can predict the systematic biases in the recalled value as observed in many empirical studies. One assumption of the model by Huttenlocher et al. is that systematic biases in response outcomes hinge on the retrieval of an anchor that contains representative information of the actual stimulus, and the location of anchor is sensitive to previous encoding of similar stimuli in memory.

In this article, I will extend previous findings by studying how the combination of external and internal representations of perceptual stimuli may lead to systematic response biases in an interactive environment. In an interactive environment, a person has to explicitly deploy an information-access action (e.g., a mouse click or key press) to perceive and encode a stimulus in the external world. Previous research (Ballard et al., 1997; Carlson, Avraamides, Cary, & Strasberg, 2007; Fu & Gray, 2006; Gray & Fu, 2004; Gray et al., 2006; O’Hara & Payne, 1998) has shown that the reliance on accessing information from the external world or from the internal memory is sensitive to the interaction cost (i.e., how effortful it is to execute the information-access action), and the resulting mix of internal and external representations of objects in the world will change as the interaction cost increases. What is still unclear is how the different mixes of internal and external representations, which act as inputs to subsequent processes, will lead to systematic biases in final response outcomes when people are interpreting, deciding, or judging the external stimuli.

2. Context effects in choice and judgment

Research has repeatedly shown that judgment and choice outcomes depend on the specific set of alternatives in which a stimulus is considered (Huber & Puto, 1983; Simonson, 1989; Simonson & Tversky, 1992). The same stimulus may be considered more desirable in one context and less desirable in another context. In fact, because explanations of context effects often involve assumptions of complex interactions between representations and processes, context effects have been repeatedly used as benchmark tests to evaluate theories (Roe, Busemeyer, & Townsend, 2001). Two most widely studied contexts were used in the experiments. At the outset, it is important to point out that the purpose of many previous studies on these context effects was to understand the underlying preference functions in a judgment and a choice task. In contrast, in the current study, these contexts were chosen to test how the effects of interaction costs on higher-level cognitive representations and processes would manifest themselves through predictable changes in behavioral outcomes in these contexts.

2.1. The similarity context

The similarity context refers to situations where a person considers stimuli X, S, and Y as shown in Fig. 1. Suppose the person is deciding between X and Y. X is high on Dimension 2 and low on Dimension 1, and Y is high on Dimension 1 but low on Dimension 2. Assume that the person considers X and Y, S and X, and S and Y equally attractive when only two stimuli are presented. One common finding is that when all three stimuli are presented at the same time in a choice task, there is a tendency that the dissimilar stimulus (Y) will be chosen more often than the other two similar stimuli (X and S). One explanation is that when the three stimuli are presented together, there is a tendency for people to select from the response categories between the similar stimuli (X and S) and the dissimilar stimulus (Y) to reduce representational complexity (Nosofsky, 1992). During response selection, when the similar category is selected, X and S will be chosen equally; but when the dissimilar category is selected, Y will be chosen (Parducci & Perrett, 1971; Wedell, 1998). Given that the probabilities for selecting between the two categories are likely to be equal, the overall effect is that the dissimilar stimulus will be chosen more often.

Figure 1.

 A graphical description of the context effects.

One implication for this explanation is that the similarity effect (higher probability of choosing the dissimilar stimulus) is dependent on the response selection process in a decision-making task, in which people are forced to make a choice among the three stimuli. If the effect is truly dependent on the response selection process, one may expect that the similarity effect will disappear when people are asked to rate each stimulus individually. If, however, the similarity effect is independent of the response selection process, but depends on the different representations of the stimuli (e.g., that the dissimilar stimulus is perceived to be more attractive), then one may expect that the same similarity effect will be observed regardless of whether it is a choice or judgment task.

2.2. The dominance context

The dominance context refers to situations when a person considers stimuli X, A, Y as shown in Fig. 1. A is dominated by X (X is better in both dimensions). As in the similarity context, X and Y are equally attractive when the two stimuli are presented together. However, when X, A, and Y are presented together and the person is asked to select the best stimulus, the probability of choosing X increases (e.g., Huber & Puto, 1983; Parducci & Perrett, 1971; Simonson, 1989). One explanation is that the inclusion of the dominated stimulus (A) has extended the range of values in the dimension of X that has a lower value than Y (i.e., Dimension 1 in Fig. 1), and the weight given to this dimension is lowered. This is because before A is added, X has the lowest value in Dimension 1; after A is added, X no longer has the lowest value in Dimension 1, and in fact it is perceived as “not as bad as A” in Dimension 1. The extension of the range of values in Dimension 1 therefore makes the overall subjective value of X higher than the nondominating Y, thus increasing the probability of choosing X (e.g., Gravetter & Lockhead, 1973; Parducci & Perrett, 1971; Wedell, 1998). The implication of this explanation is that this dominance effect is dependent on the biased representations of the stimuli and is independent of the response formats. In other words, if this explanation is correct, then one will expect that the same effect will be observed in a choice and a judgment task.

In both context effects, systematic biases in response outcomes are explained by the interactions between different representations of the stimuli (i.e., representing the stimulus as similar vs. dissimilar and dominating vs. dominated) and the response selection processes. Given that the two context effects have distinct predictable outcomes in choice and judgment tasks, they are ideal for testing the dynamics involved in the interactions between different representations and processes induced by the characteristics of the interactive environment. I will first describe the experiments that tested how changes in interaction costs would influence the choice and judgment of stimuli in the similarity and dominance context. I will then describe the dynamic context model to explain the experimental results.

3. Experiments

Two experiments were designed to test the context effects in interactive environments with different interaction costs on response outcomes in a decision-making (Experiment 1) and a judgment (Experiment 2) task. Comparisons of results from the two experiments could provide useful information on how different representational inputs would interact with different response selection processes as predicted by the model. In each experiment, the dimensions of the stimuli were represented by basic physical properties of displayed objects (line length and angle) that are representative of many interactive displays. Specifically, participants were presented with a set of squares; each of them had a line emanating from its center (see Fig. 2). Participants were asked to select the stimulus with the highest combined value represented by the length of the line and the angle that the line forms with the vertical axis (actual manipulation will be described next). This displayed object was used because line lengths and angles are elementary object properties that are commonly used to represent either separate dimensions of information or parts of more complex objects in various domains such as science, engineering, and business (e.g., Wickens & Hollan, 2000). The two dimensions are also commonly found in both separate and configural displays, in which the values from each dimension may need to be attended to separately or combined depending on information needs.

Figure 2.

 A sample screenshot of the experiment for the low-cost display in the testing stage. In the training stage, only two squares were shown. In the medium- and high-cost conditions, only the squares were shown. The squares were randomly located in one of the four quadrants throughout the experiment.

3.1. Experiment 1

In the experiment, the between-participant variable was interaction costs (low, medium, and high) and the within-participant variable was the two sets of contextual stimuli (dominant or similarity). In the low-cost display, participants could see all stimuli in the current choice set simultaneously. In the medium-cost display, only squares were displayed. Participants used the mouse to point and click on one of the squares, and the line would be shown immediately. The high-cost display was similar to the medium-cost display, except that after participants clicked on a square, there was a 2-s lockout time before the line was shown. These costs were comparable to the system response times imposed by certain interactive displays. In both the medium- and high-cost displays, as soon as participants released the mouse button, the line would disappear.

3.1.1. Method

3.1.1.1. Participants  Forty-five participants from the University of Illinois participated in this study (age: M = 22.4; SD = 2.4; 21 men). Participants were randomly divided into three groups and assigned to the low-, medium-, and high-cost conditions. Participants received $8 for participating in the experiment that lasted for about an hour.

3.1.1.2. Stimuli  Participants were given a training stage to learn to compare across dimensions and the possible ranges of the dimensional values. Participants were presented with either two (during training) or three (during testing) 5 mm × 5 mm squares located at the centers of the four quadrants of a 17-in. computer screen (Fig. 2). The actual locations of the squares in a given trial were randomized. Participants were told to always judge the angle that was smaller than 180°. There were nine equally divided levels in each dimension. Based on results from a pilot study, the lines varied in length from 20 mm to 60 mm, with each successive line length increased by 5 mm. The angles varied from 55° to 135°, with each successive angle increased by 10°. The frequency of presentation for each combination of the two dimensions was equal for participants in all conditions.

3.1.1.3. Design  Three levels of interaction costs as a between-participant variable and two contexts were used as a within-participant variable. The main dependent variables were the choice proportions of the stimuli in the two contexts.

3.1.1.4. Procedure  There were 144 trials in the training stage. In 24 of them, one stimulus was dominated by the other stimulus, so that both the line length and the angle of one stimulus were larger than the other stimulus. These dominant trials were designed to make sure that participants had experienced the dominance relationship during the training stage. In the other 120 “tradeoff” trials, if one square A had a longer line than another square B, the line in square B would make a larger angle than that in square A, or vice versa, but one of them would have a higher value than the other. The order of the dominant and tradeoffs trials and the locations (any of the four quadrants) of the squares were randomized. Instructions were given to the participants before the experiment began. When the experiment began, participants were told that there were two stages in the experiment, with 144 trials in each stage. Participants could take a short break after the first stage. When the first (training) stage began, participants were given an example in which the line lengths and angles of two squares were 30 mm, 105° and 50 mm, 45°, respectively. The experimenter would stay with the participant to answer his or her questions during the example.

In the low-cost condition, stimuli were presented on the screen simultaneously. Participants could spend as much time as they needed to inspect the stimuli, and they could then press the spacebar to make a response. Participants were told to select the stimuli that had the highest combined value (longest line and largest angle) by weighing the two dimensions equally. Responses were self-paced and were entered through the numeric keypad of a standard keyboard. Participants were told to press one of the four buttons (1, 3, 7, 9) on the numeric keypad that congruently mapped to the spatial locations of the four quadrants of the screen. If they pressed the key that corresponded to a location that did not have a stimulus, a beep sound would be given and they had to repeat the response again.

In the medium- and high-cost conditions, only squares were presented on the screen when a trial began. One of the squares was randomly selected by having a yellow circle surrounding the square. Participants were told to click on the square that was highlighted by the yellow circle, and the yellow circle would disappear and the line would appear (after a 2-s delay in the high-cost condition). As long as the participant kept the mouse button held down, the line would stay on the screen, but as soon as the mouse button was released, the line disappeared. After the mouse button was released, the yellow circle would highlight another stimulus. Participants could then click on the highlighted stimulus. After all stimuli had been studied at least once, no stimulus would be highlighted. This was to ensure that each of the stimuli was examined at least once before a response was made. At this point, participants could choose to spend as much time as they wanted to examine the stimuli again, or they could at any time press the spacebar to give a response. Participants were again told to press one of the four keys (1, 3, 7, 9) on the numeric keypad to select one of the stimuli on the screen.

During the training stage, feedback was given at the end of a trial. Specifically, the stimulus selected by the participant would turn red, and a yellow circle would highlight the correct answer. Participants could spend as much time as needed to examine the feedback. They could then press the spacebar to proceed to the next trial. Participants were informed when the training stage ended. They were also told that in the next (testing) stage, the task would stay the same except that there would be three stimuli and no feedback would be given.

In the testing stage, three stimuli would be presented. The line lengths and angles of the three stimuli were constructed based on the two context effects as depicted in Fig. 1. The stimuli used for the two contexts were shown in Table 1. All stimuli in the testing stage had been observed equally often in the training stage. There were 18 blocks in the testing stage and 8 trials per block. Within each block, choice sets from each of the context effects were randomly selected. No feedback was given during the testing stage.

Table 1. 
Stimuli used in the testing stage
XYSA
  1. Note: Refer to Fig. 1 for the meanings of X, Y, S, and A.

25 mm, 125°55 mm, 65°50 mm, 75°20 mm, 115°
55 mm, 65°25 mm, 125°30 mm, 115°50 mm, 55°
30 mm, 115°50 mm, 75°45 mm, 85°25 mm, 105°
50 mm, 75°30 mm, 115°35 mm, 105°45 mm, 65°
25 mm, 105°45 mm, 65°40 mm, 75°20 mm, 95°
45 mm, 65°25 mm, 105°30 mm, 95°40 mm, 55°
30 mm, 95°40 mm, 75°45 mm, 65°25 mm, 85°
40 mm, 75°30 mm, 95°25 mm, 105°35 mm, 65°

3.1.2. Results and discussion

At the end of the training stage, all participants attained an accuracy of at least 95% for both the dominant and tradeoff trials. There was no significant correlation between accuracy and values in either dimension. During training, there was no significant difference in mean accuracies across all conditions. However, the mean number of information accesses (clicking on a square) was significantly larger in the medium-cost condition (training: M = 6.32, SD = 1.85; testing: = 8.21, SD = 2.2) than that in the high-cost condition (training: = 3.14, SD = 0.95; testing: = 4.13, SD = 1.92) during training (t(14) = 3.82, p < .01) and testing (t(14) = 3.90, p < .01). The interaction between interaction costs and stages was not significant. The results were consistent with the prediction that participants in the high-cost condition relied more on memory-based comparisons than those in the medium-cost condition.

3.1.2.1. The similarity context Fig. 3a shows the mean choice proportions of the three stimuli in the similarity context. anova on the choice proportions of the dissimilar stimulus (Y) showed that the main effect of interaction costs was significant (F(2, 42) = 5.22, MSE = 0.134, p < .01, η2 = .20), and simple comparisons showed that all differences across different levels of interaction costs were significant (p < .05). However, this main effect was not significant for the other two similar stimuli (X and S). The choice proportion of the dissimilar stimulus was lowest in the low-cost condition and increased as interaction cost increased in the medium- and high-cost conditions. Only in the medium- and high-cost conditions, the choice proportion of the dissimilar stimulus (Y) was significantly higher than the two similar stimuli (p < .05) in the similarity context. Thus, the similarity effect (that the dissimilar stimulus was chosen more often than the two similar stimuli) was only found in the medium- and high-cost conditions, but not in the low-cost condition.

Figure 3.

 Choice proportions of each of the three stimuli in the (a) similarity and (b) dominance contexts. Refer to Fig. 1 for the definitions of X, Y, S, and A.

3.1.2.2. The dominance context Fig. 3b shows the choice proportions of the three stimuli in the dominance context. anova on the choice proportions of the dominating stimulus (X) showed that the main effect of interaction costs was significant (F(2, 42) = 6.78, MSE = 0.12, p < .01, η2 = .24). This main effect was also significant for the choice proportions of the dominated stimulus (A) (F(2, 42) = 5.82, MSE = 0.32, p < .01, η2 = .14), but this main effect was not significant for the other, nondominating stimulus (Y). The choice proportion of the dominating stimulus (X) was lowest in the low-cost condition, and it increased as interaction cost increased in the medium- and high-cost conditions. In contrast, the opposite trend was observed for the dominated stimulus, which was highest in the low-cost condition and decreased as interaction cost increased. In all three conditions, the choice proportions of the dominating stimulus were significantly higher than the other two stimuli (p < .05). Thus, the dominant effect was observed in all three conditions.

To summarize, results from Experiment 1 showed that as interaction costs increased, choice proportions of the dissimilar stimulus in the similarity context and the dominating stimulus in the dominance context increased. This finding provides support for the hypothesis that as interaction cost increases, the shift from deictic to memory representations of the contextual stimuli leads to systematic biases in decision outcomes. To understand how the shift in representations may lead to changes in decision outcomes, a dynamic context model is developed. However, before describing the details of the model, I will first present results from the second experiment that provide further constraints to the mechanisms of the model.

3.2. Experiment 2

In Experiment 2, the same materials in Experiment 1 were used, but the task was changed from a decision task to a judgment task. The major difference between a decision task and a judgment task was that a decision was explicitly comparative in nature but a judgment typically was not. A second difference was that responses in a judgment task were elicited using a graded scale, but in a decision-making task all-or-none responses were elicited. Comparisons of decision-making and judgment outcomes would therefore allow further testing of whether the hypothesized effects were sensitive to different response formats imposed by the task.

Although results from the similarity context of Experiment 1 supported the hypothesis that increase in interaction costs will lead to systematic biases in choice outcomes, it could not rule out the possibility that the dissimilar stimulus was perceived to have a higher combined value than the similar stimuli as interaction cost increased. For example, one might argue that a higher interaction cost could lead to a higher weight given the dimension that the dissimilar stimulus had a higher value, making the perceived combined value of the dissimilar stimulus higher. If this was the case, one would expect that the dissimilar stimulus would be judged to have a higher rating than the similar stimuli. However, if the higher choice proportions of the dissimilar stimulus depended on the all-or-none choice process, one would expect that the dissimilar stimulus would be given equal ratings as the similar stimuli in a judgment task.

In the dominance context, results from Experiment 1 were again supportive of the hypotheses that the higher interaction cost had induced more reliance on memory representations of stimuli. If the overall subjective value of the dominating stimulus was assumed to be higher, one would expect that it would also be given a higher rating in a judgment task. However, it was possible that the observed higher choice proportions of the dominating stimulus were specific to the response formats imposed by the choice task. For example, it was possible that the higher interaction cost did not enhance the range effect. Instead, participants might have eliminated the dominated stimulus early on during the encoding, without changing the perceived value of the dominating stimulus. In addition, because choosing the dominating stimulus could be more easily justified than choosing the non-dominating stimulus (Simonson, 1989), the higher choice proportions of the dominating stimulus did not necessarily reflect the fact that the subjective value of the dominating stimulus was higher. If this was true, one would expect that similar ratings would be given to the dominating and nondominating stimuli in a judgment task.

3.2.1. Method

3.2.1.1. Participants  Forty-five participants from the University of Illinois participated in this study (age: M = 21.7; SD = 2.1; 23 men). Participants were randomly divided into three groups and assigned to the low-, medium-, and high-cost conditions. Participants received $8 for participating in the experiment that lasted for about an hour.

3.2.1.2. Design  The same independent variables were used in Experiment 2: three levels of interaction costs as a between-participant variable and two contexts were used as a within-participant variable. The main dependent variables would be the choice proportions of the stimuli in the two contexts. The same set of stimuli in Experiment 1 was used in Experiment 2.

3.2.1.3. Procedure  The procedure was similar to that of Experiment 1. However, instead of selecting a stimulus, participants were asked to rate each stimulus by combining the two dimensions, and they gave a response using a numerical scale from 1 to 9 (with 1 for the combination of the shortest line and smallest angle, 9 for the longest and largest angle). Participants were told that the two dimensions had equal weight. After an example trial (same as that in Experiment 1), participants were given the same set of 144 trials in the training stage, and another 144 trials in the testing stage as in Experiment 1. When a trial began, all stimuli were presented on the screen. In the low-cost condition, all lines were shown on the screen. Participants could spend as much time as needed to inspect the stimuli, after which they could press the spacebar to make a response. In the medium- and high-cost conditions, only squares were presented, and participants had to click on a square to show its line. When a trial began, one of the squares was randomly selected by having a yellow circle surrounding the square. Participants were told to click on the square that was highlighted by the yellow circle, and the yellow circle would disappear and the line would appear (after a 2-s delay in the high-cost condition). After the mouse button was released, the yellow circle would highlight another stimulus. Participants could then click on the highlighted stimulus. After all stimuli had been studied at least once, no stimulus would be highlighted. At this point, participants could choose to spend as much time as they wanted to examine the stimuli again, or they could press the spacebar to give a response. After the spacebar was pressed, participants would be asked to click on one of the stimuli to select it, and enter a rating from 1 to 9 for the selected stimulus using the numeric keypad. This process continued until all stimuli were rated. After all ratings were given, the correct ratings would be displayed next to each of the stimulus to provide feedback to participants’ performance. Participants could spend as much time as needed to examine the feedback. They could then press the spacebar to proceed to the next trial. Participants were informed when the training stage ended. They were also told that in the next (testing) stage, the task would stay the same except that there would be three stimuli and that no feedback would be given. The same set of stimuli in Experiment 1 was used in Experiment 2.

3.2.2. Results and discussion

At the end of the training stage, the subjective ratings given by participants were close to the objective ratings, as measured by the mean absolute differences (MAD) between them (M = 1.39, SD = 0.92). No correlation was found between the means and variances of the absolute differences and objective values in either dimension. There was no significant difference in the measure of MAD for interaction costs. However, similar to Experiment 1, the mean number of information accesses was significantly larger in the medium-cost condition (training: = 5.32, SD = 0.85; testing: = 6.21, SD = 1.2) than that in the high-cost condition (training: = 3.14, SD = 0.65; testing: = 4.13, SD = 0.92) during the training (t(14) = 3.91, p < .01) and testing (t(14) = 5.70, p < .01) stages. The interaction between interaction costs and stages was not significant. Participants in the high-cost condition checked the stimuli less often than in the medium-cost condition and relied more on memory-based operations during the judgment process.

3.2.2.1. The similarity context Fig. 4a shows the mean ratings of the three stimuli in the similarity context. Separate anovas on the ratings of each of the stimuli showed that there was no significant effect of interaction costs in any of the stimuli. There was also no significant difference in the mean ratings for the stimuli in any of the interaction cost condition. Unlike results from Experiment 1, the dissimilar stimulus was given the same subjective rating as the other two stimuli. The results supported the explanation that the higher choice proportion of the dissimilar stimulus was because of the all-or-none response format imposed by the choice task in Experiment 1. When the response was to give ratings to each individual stimulus in Experiment 2, this effect disappeared.

Figure 4.

 Judgment ratings of each of the three stimuli in the (a) similarity and (b) dominance contexts. Refer to Fig. 1 for the definitions of X, Y, S, and A.

3.2.2.2. The dominance context Fig. 4b shows the mean ratings of the three stimuli in the dominance context. In general, the pattern of results was similar to that in Experiment 1. anova on the ratings of the dominating stimulus (X) showed that the main effect of interaction costs was significant (F(2, 42) = 15.87, MSE = 3.21, p < .01, η2 = .32). This main effect was also significant for the ratings of the dominated stimulus (A) (F(2, 42) = 8.97, MSE = 0.952, p < .01, η2 = .12), but this main effect was not significant for the other, non-dominating stimulus (Y). Similar to results from Experiment 1, the rating of the dominating stimulus (X) was lowest in the low-cost condition and increased as interaction cost increased in the medium- and high-cost conditions. In contrast, the opposite trend was observed for the dominated stimulus, which was highest in the low-cost condition and decreased as interaction cost increased. In all three conditions, the rating of the dominating stimulus was significantly higher than the other two stimuli (p < .05) in the dominance context. The results provided support for the explanation that the dominance relationship increased the subjective value of the dominating stimulus, and this effect was enhanced with the increase in interaction costs.

3.3. Summary of results from the two experiments

Results from the two experiments provided support to the hypothesis that changes in interaction costs lead to a shift from deictic to memory representations, which subsequently induce systematic biases in decision-making and judgment outcomes. In the decision-making task in Experiment 1, the dissimilar stimulus was selected more often than the similar stimuli, and this difference increased as interaction cost increased. In Experiment 2, when participants were instructed to assign judgment ratings to the stimuli, no significant difference was found in the ratings between the stimuli in the similarity context, supporting the notion that the represented values of the similar and dissimilar stimuli were the same. The results confirmed that the shift from deictic to memory representations of the stimuli did not change the recalled values in the similarity context. The systematic biases observed in the choices of stimuli were therefore likely caused by the difference in response selection process. The pattern of results therefore demonstrated how changes in interaction costs lead to changes in the response selection processes.

In contrast to the similarity context, in both experiments, similar response biases were observed in the dominance context. In Experiment 1, the dominating stimulus was chosen more often; in Experiment 2, the dominating stimulus was given a higher rating than other stimuli. The systematic biases in the dominance context in both experiments showed that they were independent of the response formats. The results were consistent with the hypothesis that the dominating relationship led to a range effect that boosted the value assigned to the lower-valued dimension of the dominating stimulus, which eventually increased the combined value of the dominating stimulus but not the nondominating stimulus. As interaction cost increased, the systematic bias was strengthened, suggesting that the range effect was enhanced as representations shifted from deictic to memory-based. The pattern of results therefore demonstrated that increase in access costs led to changes in cognitive representations of the contextual stimuli.

To provide a deeper understanding on how the change in interaction costs led to different representations and processes in the similarity and dominance contexts, a computational cognitive model was developed to explain the patterns of results from both experiments.1 The goal of the model was to provide a mechanistic explanation to the effects of interaction costs on encoding strategies, representations of the two sets of contextual stimuli, and the response selection processes in both contexts, and how they would produce the patterns of results in both Experiments 1 and 2.

4. The dynamic context model

Based on previous research (Ballard et al., 1997; Fu & Gray, 2006; Gray & Fu, 2004; Gray et al., 2006), the current model assumes that there is a mix of memory and perceptual-motor operations that encode the stimuli in the interactive environment. Consistent with the theory of soft constraints (Gray et al., 2006), the precise mix depends on the costs and benefits of information accesses. However, as will be discussed next, the model predicts that the moment-to-moment cost–benefit tradeoffs will induce different cognitive representations and processing of the set of stimuli, which will eventually influence the response selection processes and outcomes. I will first provide a general description of the main components of the model before I describe their implementation with respect to the specific setups in the experiments in the next sections. The predicted effects of the model at different levels of processing are summarized in Table 2.

Table 2. 
The predicted effects of the increase in interaction costs on the use of the processing operator, encoding strategy, cognitive representation, and response selection process
Interaction CostLowHigh
  1. Note: Predictions of the behavioral outcomes in the two contexts are listed (E1, Experiment 1; E2, Experiment 2). The arrows indicate the predicted direction of shift as interaction cost increases.

OperatorPerceptualMemory
Encoding strategy and cognitive representationDeicticMemory
Response selectionSelect stimulus based on perceptual comparisonsDecision
Select stimulus based on memory comparisons
Generate ratings based on perceptually integrated dimensional valuesJudgment
Generate ratings based on combined values recalled from memory
Context effectsX and S perceived to be less similar because of deictic representations and perceptual re-encoding and comparisonsSimilarity
X and S perceived to be more similar because of memory representations and memory comparisons
Range effect increases perceived value of the dominating dimensionDominance
Range effect enhanced by re-adjustments of dimensional and combined values of stimuli
E1 predictions (choice task)X, S, Y equally probable to be selectedSimilarity
(X, S) and Y likely recalled from memory and compared; Y is more likely selected than X and S.
X is more likely selected because of its higher perceived valueDominance
X is even more likely selected because the dominated stimulus had a even lower value because of re-adjustments
E2 predictions (judgment task)X, S, Y are given similar ratingsSimilarity
X, S, Y recalled from memory are given similar ratings
X is more likely given a higher rating than YDominance
Recalled memory representation of X is more likely given a higher rating than that of Y

4.1. Encoding strategies and representations

It was hypothesized that an increase in access costs would induce different cognitive encoding strategies and representations of the stimulus. Ballard et al. (1997) argued that when access cost was low, people tended to adopt a deictic encoding strategy and representation, in which the external environment serves as a stable source of external memory, and perceptual processes (such as eye fixations) are effectively utilized to acquire information on a just-as-needed basis from the external environment to minimize use of working memory. In other words, instead of encoding the value of a dimension, a deictic representation encodes only the pointer to the location where the information can be found in the external environment. When the value of a specific dimension is needed, a perceptual routine will be deployed to acquire that precise value (see Fig. 5).

Figure 5.

 A notational diagram illustrating the hypothesized effects of interaction costs on encoding strategies. Cat(X) represents the categorical rating of X.

Based on previous results, the current model assumed that as interaction cost increased (e.g., from an eye movement to a mouse click), memory encoding and representations of dimensional values became relatively less costly and were preferred over the deictic encoding and representations (Fu & Gray, 2000; Gray et al., 2006). After the stimulus was encoded in memory, values of the stimulus recalled from memory would depend not only on the encoded values but also on the categorical ratings of the stimulus (Huttenlocher et al., 1991; Parducci & Perrett, 1971). Results from these studies implied that memory representations could introduce systematic biases in the recalled values of perceptual stimuli, and these biases may be sensitive to the encoded categorical information.

In the model, when either the line length or angle of a stimulus A was perceptually encoded, the physical magnitude S was converted to a psychological magnitude P(S) by the following equation (e.g., Ekman, 1958; Stevens, 1975):

image(1)

in which kD was a coefficient of proportionality and was set to a constant of 1, and σ (with M = 0 and SD = 0.01) represented noise in the conversion process. The model therefore assumed a simple Gaussian noise in the perceptual judgment of the stimulus magnitude.

In the memory strategy, the perceptually encoded P(S) would be stored in a slot of a memory chunk representing the stimulus A. Based on results from previous studies (e.g., Huttenlocher et al., 1991), it was assumed that a categorical coding would be encoded together with the stimulus into memory. Specifically, it was assumed that the continuous stimulus value P(S) would be converted linearly to a memory anchor, Cat(S), which ranges from 1 to 9, which matches the levels in the original stimuli.

The current model assumed that as contextual stimuli were encoded, the categorical rating assigned to each dimensional value would be influenced by the previous ratings through an anchor and adjustment heuristic (e.g., Kahneman & Tversky, 1974). Fig. 6a shows how the model encoded stimuli in the similarity context using deictic representations. In the actual simulation, the model would randomly pick a dimension of one of the stimuli to encode in each cycle, but for the sake of clarity, values of both dimensions of the same stimulus were encoded consecutively in Fig. 6a. In this example, the model began with stimulus X. The values of Dimensions 1 and 2 of X were encoded with ratings of 5 and 1, respectively. It was assumed that when the second stimulus was selected, each of the dimensional values would be compared with the last encoded ones. When the difference of the categorical ratings between two stimuli was small, the anchor and adjustment heuristic would be applied. In this example, S was selected next, and because the dimensional values were judged to be close enough to those of X, the anchor and adjustment heuristic was applied. Specifically, the dimensional values of S were compared with those of X, and the resulting categorical ratings were then encoded. Finally, because the dimensional values of Y were quite different from S, the heuristic was not used, and the categorical ratings of 1 and 5 were encoded. Note that because both the perceptual judgment and the memory retrieval process were noisy, the resulting categorical ratings could be different from the true values.

Figure 6.

 The adjustment heuristic used by the model during the encoding process. In actual simulations, the model would randomly pick a dimension of one of the stimuli to encode in each cycle when the deictic strategy was used. When the memory strategy was used, adjustment could be triggered by either individual dimensional values or the combined values.

During memory encoding, the same anchor and adjustment process will be performed. As shown in Fig. 6, the categorical values of X were first encoded as 5 and 1, and the sum of the values, in this case 6, were encoded together with the dimensional values and stored in the memory representation of the stimulus. When the model encoded S, it would retrieve X from memory and compared S with the retrieved chunk X. In this case, only the difference between the categorical ratings of the dimensional values would be considered. When this difference was smaller than 2, the values of S would be adjusted to 4 and 2, and the sum of the values, 6, was encoded in chunk S. Finally, the dimensional values for chunk Y would be 1 and 5, and the sum would be 6. One major difference between the deictic and memory strategy was that, in the memory strategy, only categorical values were retrieved to trigger the anchor and adjustment process, but in the deictic strategy, both the categorical values and the perceived dimensional values were available when comparing between stimuli.

The model used the memory representations and retrieval mechanism in the adaptive control of thought-rational (ACT-R) architecture (Anderson et al., 2004) to represent the anchors. In ACT-R, memory elements are represented as chunks, and each chunk contains slots that store values of its element. For example, a chunk for anchor i can have slots storing the psychological magnitudes (i.e., P(S)) of the line length and the angle that the line makes with the horizontal, as well as their categorical coding (i.e., Cat(S)). A chunk i has an activation value Ai that determines how likely it is that it can be retrieved. In the current model, when it requests to retrieve an anchor chunk that has a particular set of slot values, Ai can be calculated by2

image(2)

In Eq. 2, Bi represents the base-level activation of the chunk and Mli represents the degree of dissimilarities of slot l of chunk i and the slot value of the requested chunk to be retrieved, and σ represents Gaussian noise with a mean of 0 and unit variance. The base-level activation Bi reflects the recency and frequency of accesses to the chunk and it is calculated as:

image(3)

in which tj represents the time lags from the present after the chunk has been used, and d is a decay parameter. The base-level activation therefore captures the history of use of a chunk by the logarithm of the sum of its individual uses that decays as a power law. In general, chunks that are used recently and frequently tend to have higher base-level activation, but the activation decays with time as the chunk is not used again. When activation of a chunk decays below the activation threshold (which has a default value of 0), retrieval fails.

The degree of dissimilarities between the category ratings of i and j (i.e., Mli) was set using a linearly proportional value from −2.0 (most dissimilar) to 0.0 (identical). Because the categorical ratings ranged from 1 to 9, the maximum difference between the two values was 8 (which corresponded to a Mli value of −2.0), and a difference of 7 would lead to a Mli value of −1.75, and so on. The summation of the dissimilarities in Eq. 2 implies that if the slot values of a chunk are highly dissimilar to those of the requested chunk, the activation value of that chunk will be lower. In other words, with everything else equal, chunks that are closest to the requested chunk will mostly likely be retrieved, but the noise in the calculation of activations (Eq. 2) makes this a stochastic process. Therefore, when the model tried to determine the categorical rating of a stimulus by retrieving the anchor from memory, the retrieved anchor might or might not be the one that matched exactly the requested chunk. The stochasticity involved in the retrieval process was determined by Eqs. 2 and 3. It is important to point out that, when a deictic representation was used, the category rating would be derived directly based on the encoded value P(S), and was therefore not subject to the same noisy memory retrieval process.

4.2. Decision strategies

Fig. 7 shows the flowcharts of the two possible decision strategies the model would use in Experiment 1. Similar to the different mixes of perception- and memory-based operations induced by different interaction costs, the deictic and memory decision strategies are meant to represent two ends of the extremes with different mixes of the two strategies in between (in fact, it is unlikely that one would adopt a purely deictic or a memory strategy). For example, the actual strategy used may vary from a deictic strategy that does not encode any value into memory to a partial memory strategy that encodes only one stimulus into memory or a memory strategy that encodes both stimuli into memory during the comparison and decision process.

Figure 7.

 The deictic and the memory decision strategies. The two strategies represented two ends of the extremes of the set of all possible strategies with different mixes of deictic and memory encoding and representations.

The current hypothesis is that an increase in interaction costs will induce a shift from perception- to memory-based processes, which in turn will induce a shift from a deictic to a more memory-intensive encoding strategy. The actual strategy adopted by the model depended on both the characteristics of the environment (whether it was the low-cost or the medium-/high-cost condition) and the availability of the different cognitive representations of the stimuli (e.g., whether the model could retrieve an encoded stimulus from memory). For example, stimuli were all visible in the low-cost experimental condition but in the medium- and high-cost conditions, participants needed to click on a stimulus to see the dimensional values and only one stimulus could be seen at a time. Therefore, the deictic strategy described in Fig. 7a was only possible in the low-cost condition, where individual dimensions of two stimuli could be perceptually compared. When only one stimulus was visible at a time, the comparison had to be performed by encoding at least one stimulus in memory.

In the deictic decision strategy, two stimuli would first be randomly selected from the environment, and each dimension would be separately encoded (and its categorical rating assigned) and compared to determine if one stimulus was better than the other (the actual comparison process will be discussed next). In the memory strategy (Fig. 7b), one stimulus would be selected, and both dimensions of the stimulus would be encoded into the memory chunk. The model would then attempt to retrieve another stimulus from the memory, and, at the same time, try to access a stimulus by clicking on it.3 In other words, the retrieval process and the process of accessing a stimulus in the environment competed with each other, and the process that finished first won. As the interaction cost increased from the medium- to the high-cost condition, the likelihood for memory retrieval to finish first increased. Hence, there would be more comparisons of stimuli in memory in the high-cost than in the medium-cost condition.

Two stimuli were compared based on the encoded perceived dimensional values and their categorical ratings. In the deictic strategy, stimulus X would be considered better than stimulus Y when the sum or the perceived value of X was larger than that of Y. Otherwise, X would be selected when

image(4)

In Eq. 4, PD1(X) represented the encoded value of Dimension 1 of stimulus X, and PD2(Y) represented the encoded value of Dimension 2 of stimulus Y, and so on. w was a random variable that controlled the weight given to one dimension (line length), and it followed a uniform random distribution defined in the range of (0, 1). The randomness represented the attentional shifts given to any one dimension within a single trial during the decision process (e.g., Busemeyer & Townsend, 1993). H was the threshold value that controlled if the sum of the differences in both dimensions was large enough to lead to the decision that X was better than Y, and was set to a value of 1.5. As shown in Fig. 7, when the threshold was not reached, the model would pick another stimulus, re-encode it, and compare the dimensions again until one stimulus could be eliminated (this was possible because of encoding and retrieval noise). The same process would continue until all but one stimulus were eliminated and a response could be made. As Eq. 4 shows, the criterion for eliminating a stimulus depended on both the attentional bias and the differences between the categorical values of both dimensions of the stimuli. In other words, given the same probability of an attentional bias, a stimulus would less likely be eliminated when it is compared with a similar stimulus. When the difference did not reach the threshold, another pair of stimuli would be selected until one could be eliminated.

In the memory strategy, X would be considered better than Y when the sum of the categorical values of X was larger than that of Y. Otherwise, X would be selected when

image(5)

In Eq. 5, CatD1(X) represented the categorical rating of Dimension 1 of X, and CatD2(Y) represented the categorical rating of Dimension 2 of Y, and so on. The definitions and values of w and H were the same as in Eq. 4. Note that the only difference between Eqs. 4 and 5 was that, when the deictic strategy was used, the model could perceptually compare the stimuli; but when the memory strategy was used, the model had to rely on the retrieved categorical values of the stimuli.

4.3. Explaining the response biases in the experiments

I will now discuss how the combination of the anchor and adjustment heuristic and the selection of decision strategies would lead to the response biases in both experiments. The same model was run to fit the data in the two contexts in both experiments. The main goal of the model was to show how an increase in interaction cost would lead to a change in encoding strategies, cognitive representations, and decision strategies, and how they explain the patterns of results in the two experiments.

4.3.1. Response bias in the similarity context in Experiment 1

When the model began, it would first encode the three stimuli according to either the deictic or memory strategy (predominantly the deictic strategy in the low-cost condition, and increasingly more use of the memory strategy in the medium- and high-cost conditions). After the first round of encoding and the adjustment of categorical ratings, as shown in Fig. 6, the model would start to decide on a response as shown in Fig. 7.

Fig. 8 shows the three possible pairwise comparisons among the stimuli in the similarity context of Experiment 1. Using the same notations as in Fig. 1, the two similar stimuli were denoted as X and S (X had a higher value in Dimension 1; S had a higher value in Dimension 2), and the dissimilar stimulus was denoted by Y. There could be three possible cases in the first comparison: Y versus S, Y versus X, and X versus S.

Figure 8.

 The three possible pairwise comparisons performed by the model for the deictic and memory strategies in the similarity context of Experiment 1. Bolded comparisons would most likely lead to a winner. Comparisons with smaller fonts were least likely to lead to a winner. When a comparison did not lead to a winner, another pair of stimuli would be selected and compared; and if only two stimuli were left, the stimuli would be re-encoded or retrieved from memory until one could be eliminated.

When the deictic strategy was used, perceptual comparisons of individual dimensions might detect the difference between X and S, and the attentional bias to D1 would lead to more selection of X than S (represented by the solid line to X and dashed line to S in Fig. 8). When the memory strategy was used, because comparison between X and S would be influenced by the noisy memory retrieval process, in which the wrong anchors that had slightly larger or smaller dimensional values than X and S could be retrieved (see Eq. 2). In addition, in the medium- and high-cost conditions, the model could not directly perceive the dimensional values of the stimuli, and therefore had to rely on the retrieved categorical values of X and S during the comparison. Thus, compared with the deictic comparisons in the low-cost condition, memory comparisons between X and S in the medium- and high-cost conditions would more likely not exceed the decision threshold. When the model could not eliminate a stimulus during a comparison, it would be forced to select another pair to compare again (and the attentional bias could be different).

As shown in Fig. 8, the attentional bias to either dimension would effectively induce two response categories, one for the dissimilar stimulus Y and the other for the similar stimuli X and S. When attention was biased to D2, the model would likely eventually select Y; but when attention was biased to D1, depending on perceptual noise, either X or S would be eventually selected (although X would be slightly more likely be selected than S). Therefore, in general, Y would be more likely selected in the similarity context (as shown by the bold path in Fig. 8), and this effect would be stronger in the memory strategy because the comparison between X and S would more likely fail to reach the decision threshold.

When the first comparison was between Y and X (second row in Fig. 8), the overall effect would be similar to that between Y and S. Specifically, attentional bias would again lead to the selection of either Y or X in the first comparison. When attention was biased to D2, Y would more likely be selected, and it would also likely be selected when subsequently compared with S. When attention was biased to D1, X would likely be selected. When X was selected, X would be compared with S. When the deictic strategy was used, the bias to D1 could lead to the selection of X. However, when the memory strategy was used, the model would less likely detect the difference between X and S. Therefore, similar to the first case, the overall effect was that Y would be more likely selected than X or S.

When X was first compared with S (third row in Fig. 8), because the two stimuli were similar and the encoding process was noisy, it was relatively unlikely that one of them would be eliminated (compared with the other two pairwise comparisons discussed before). When this happened, another two stimuli would be selected and compared again. However, the noisy encoding and the stochastic attentional biases could sometimes make one of them better than the other one, but this was more likely when the deictic strategy was used than when the memory strategy was used. As shown in Fig. 8, when the deictic strategy was used, attention could sometimes be biased to D1 (or D2), making X (or S) more likely to win when individual dimensions were compared between them. The bias to either dimension was equally likely, but if X (or S) was selected, the bias to D1 (or D2) would increase the likelihood that X (or Y) would win in the subsequent comparison. On the other hand, when the memory strategy was used, because of the noisy retrieval process, the attentional bias to D1 (or D2) would less likely lead to S (or X) to be eliminated. However, in case one of them was eliminated, the bias would play a larger role in the subsequent comparison because the difference between either dimension would be much larger (i.e., between X and Y or S and Y). Thus, if the bias was to D1 (or D2), X (or Y) would be more likely selected.

As shown in Fig. 8, when comparing between Y and X or Y and S, attentional bias to D2 would more likely lead to Y being chosen, but attentional bias to D1 would in general equally split the chance that X or S would be chosen (although the model predicted that X would be slightly preferred over S in the deictic strategy, which matched well with the empirical results in Experiment 1). Given that attentional bias to D1 or D2 was equally likely, the overall effect was that Y would be chosen more often than X or S, and this effect would be stronger when the memory strategy was used than when the deictic strategy was used. As shown in Fig. 7, when interaction cost increased, the chance that the memory strategy was used increased. The increase in the use of the memory strategy led to a higher likelihood of choosing the dissimilar stimulus than the other two stimuli. Thus, the choice proportion of the dissimilar stimulus (Y) was higher in the high-cost than the medium-cost condition. In the low-cost condition, the deictic strategy was better at detecting the difference between the similar stimuli, and thus the difference in choice proportions among the three stimuli was not significantly different.

As shown in Fig. 3a, the model simulations provided good match to the empirical data (R2 = .91). The model chose the dissimilar stimulus more often as interaction cost increased, which matched the choice proportions by the participants. Interestingly, the model also chose the similar stimulus S less often as interaction cost increased, and this matched the choice proportions of S by participants well. As shown by Fig. 3, the model chose S less often because when attention was biased to D1, X was more likely to be chosen; and when attention was biased to D2, Y was more likely to be chosen. This bias was exacerbated by the increase in interaction cost, as memory retrieval of stimuli made it harder to eliminate a stimulus when X was compared against S, and thus even decreasing the choice proportion of S.

4.3.2. Response bias in the dominance context in Experiment 1

Results from Experiment 1 showed that the dominating stimulus was chosen more often than the other stimuli. The model explained this response bias based on the assumption that the encoded categorical rating of the dominating stimulus was influenced when the dominated stimulus was encoded, and this effect was characterized by the adjust process as shown in Fig. 7b. When the deictic strategy was used, each of the categorical ratings of X was encoded. When A was selected, the value of Dimension 1 of A was compared with that of X, and a rating of 4 was encoded. However, when the value of Dimension 2 of A was compared with that of X and was found to be smaller based on the perceived value of the dimension, the model would adjust the categorical ratings of both stimuli by extending the range of the dimension. In other words, the combined value of the dominating stimulus would be boosted as a result of this range effect.

When the memory strategy was used, both dimensions of the X would be encoded together with the categorical value of the stimulus. Initially, similar to the deictic strategy, the values of 5 and 1 were encoded for X. When the model encoded A, the dimensional values of A were compared with the retrieved chunk X. Similar to the deictic strategy, the values of A were encoded as 4 and 1, and the model detected that the second dimension of A should be smaller than that of X, then a correction would be made to chunk X, such that the new encoded values for X would be changed to 5 and 2. Note that because the stimuli could not be perceived during the comparison, the adjustment process was based solely on the encoded categorical values. The overall effect was that when the memory encoding strategy was used, there would be a lower chance that a correction would be missed than when the deictic strategy was used.

As shown in Fig. 3b, the model simulations provided good match to the empirical data (R2 = .92). Because the adjustment heuristic was applied when the model converted the physical magnitudes of the stimuli to psychological ratings, the range effect influenced the representations of stimuli for both the deictic and memory encoding strategies. However, as can be seen in Fig. 3b, the choice of the dominated (dominating) stimulus decreased (increased) as interaction cost increased. As interaction cost increased, response selection shifted from predominantly perceptual comparisons of individual dimensional values to predominantly memory comparisons of the integrated dimensional values of stimuli. As shown in Fig. 7, a deictic decision strategy tended to involve more re-encoding during the perceptual comparisons of stimuli, and because of the noise, this re-encoding could sometimes lead to selection of the dominated stimulus, offsetting the range effect caused by the adjustment heuristic. On the other hand, when the memory encoding strategy was used, memory retrieval of stimuli would lead to retrieval of the combined categorical ratings of both dimensions as well as those for the individual dimensions. Because the combined ratings as well as the ratings of the two individual dimensions of the dominated stimulus were lower than those of the dominating stimulus, the chance that the dominated stimulus would be selected would be lower when the memory strategy was used. As shown in Fig. 3, the selection of the dominated stimulus was rare, but it occurred increasingly less often when the interaction cost increased, in which re-encoding became more costly and memory comparisons became dominant. As a result, selection of the dominated stimulus decreased.

4.3.3. Lack of response bias in the similarity context of Experiment 2

The same model was used and produced the results shown in Fig. 4, which shows that the response bias found in Experiment 1 was gone when the response format was changed to judgment ratings. In both experiments, the model had similar encoded categorical ratings for all stimuli (see Fig. 4) regardless of whether the deictic and memory representations were used. Therefore, when the model reported the encoded ratings, there was no significant difference between the stimuli in all three conditions.

4.3.4. Response bias in the dominance context of Experiment 2

Fig. 4b shows that the model provided good match to the empirical results (R2 = .95). Similar to the results in Experiment 1, the model explained the higher value of the dominated stimulus by the range effect created by the anchor and adjustment process. As interaction cost increased, the memory encoding strategy became dominant. The memory strategy led to a higher chance that the dominated stimulus was given a relatively lower rating compared with the other stimuli. In contrast to the results in the similarity context, because the categorical ratings of these stimuli were different, the same response bias was observed regardless of whether it was a decision or a judgment task.

5. General discussion

Results from the current set of experiments provide support to the hypothesized effects of interaction costs on the shift from perception-based to memory-based encoding strategies, and how they lead to the different mixes of the cognitive representations in the two contexts. The model also provides a good characterization of how the biased representational inputs to the response selection process may lead to systematic biases in decision-making and judgment outcomes. In both experiments, higher interaction costs induced more use of the memory representations of the stimuli. Consistent with the predictions of the dynamic context model, when participants were forced to make an all-or-none selection in Experiment 1, memory-based comparisons of the stimuli enhanced the similarity and dominance effects in both contexts. When participants rated individual stimuli in Experiment 2, response bias was found only in the dominance context, but the response bias in the similarity context disappeared.

The dynamic context model presented in this article is developed based on mechanisms in the ACT-R architecture. The model assumed that external (visual search or click on stimuli) and internal (memory retrieval) accesses of information competed with each other, and in general, accesses that took less time would win. This competition process simulated how increases in external access costs could lead to more internal accesses, and thus a shift from predominantly perceptual-motor to memory-based operations. Although this mechanism was slightly different from previous models that characterized that the shift based on a utility-based action selection process (e.g., Fu & Gray, 2006; Fu & Pirolli, 2007; Gray et al., 2006), time costs were still assumed to be the major factor underlying the decision on where to access information. On the other hand, because the current model explains the shift between perceptual and memory encoding by the competition of the corresponding parallel perceptual and memory mechanisms, the sensitivity to the interaction costs is directly tied to the architectural constraints provided by ACT-R (rather than by utility calculations, which are tied to previous experiences). The current model could therefore explain why this tradeoff between internal and external representations exists even in unfamiliar task environments (e.g., when someone is engaged in a new interaction method), with which apparently the person has no previous experiences. It is, however, possible that the precise tradeoff is a result of both architectural constraints and previous experiences. Future research on how they contribute to the tradeoff in different conditions and context will provide more insight on the nature of the tradeoff, and other factors that may influence when and how different mixes of internal and external representations would be used.

In addition to the shift of reliance from external to internal information sources, the major goal of the model was to show that different mixes of representations and processes could lead to systematic biases in response outcomes. Indeed, the model showed that as the inputs to the encoding and comparison processes shifted from deictic to memory representations, the similarity effect was enhanced in the choice task, and the dominance effect was enhanced in both the decision and judgment tasks. In the dynamic context model, similar and dissimilar response categories emerged when the encoded stimuli were retrieved and compared in memory, which led to a higher chance of selecting the similar stimuli, and thus the similarity effect was enhanced. In the dominance context, the interaction between the noisy memory retrieval process and the use of adjustment heuristic in the model led to a higher category rating of the dominating stimuli, which explained the enhanced dominance effect. These systematic biases in response outcomes were explained by the shift in the relative mixes of the external and internal representations of stimuli and processes, which defined the dynamic context in the interactive environment.

Consistent with previous findings, results from the two experiments demonstrated that when interaction costs increased, participants reduced the number of accesses of information in the external display and relied more on information encoded in memory. As a result, memory-based operations became more dominant in the judgment and decision-making processes. Previous research often concluded that an increase in memory-based operations induced by higher interaction costs would impose a higher working memory load on the operator, and thus often lead to performance declines (e.g., Wickens & Holland, 2000). Results from the current experiments have significantly broadened our understanding of the effect of interaction costs on performance in interactive environments. Specifically, the studies showed that changes in interaction costs not only induce a shift from perception-based to memory-based operations but could also lead to changes in encoding strategies and cognitive representations, which subsequently could lead to systematic biases in judgment and decision-making outcomes. The results also demonstrated that the judgment and decision-making processes are highly dynamic and adaptive to the dynamic contexts imposed by the task environment, and they argue against the view that people adopt a rigid set of top–down decision strategies or heuristics independent of the characteristics of the interactive environments.

The empirical and modeling results are in general consistent with the view of interactive cognition (e.g., Ballard et al., 1997; Wilson, 2002). The results provide support for the view that the human information-processing system is highly adaptive in selecting where to acquire the necessary information to accomplish a task. When the interaction cost is low, perception-based operations can be used to exploit the stable external information source to acquire information just when it is needed. When the interaction cost is high, memory-based operations will encode and retrieve information internally, such that there will be less reliance on external information sources. The adaptive nature of this shift from perception-based to memory-based operations suggests that the human information-processing system is highly flexible in selecting an information source that allows efficient accesses to the necessary information. The most important finding from the two experiments and the model simulations is that although the human information-processing system may seem indifferent to the source of information (Fu & Gray, 2000; Fu & Gray, 2006; Gray et al., 2006), the differences in the intrinsic nature of the processes that operate on the external and internal information sources may lead to systematic biases in behavior. Specifically, although the shift to memory-based operations can be considered an adaptive response to the increase in external interaction cost, the “side effect” of this adaptive response is that memory encoding may implicitly interact with different cognitive representations and processes, which may in turn lead to systematic biases in response outcomes in certain task contexts. The current results imply that, rather than assuming a fixed set of cognitive representations, the development of interactive cognitive models should pay more attention to the dynamics in the shift between internal and external representations as the model interacts with the environment, as they could lead to systematic differences in behavioral outcomes.

Footnotes

  • 1

     The model can be downloaded from the Cognitive Science Website.

  • 2

     This activation equation is simplified to focus on only the variables that were used in the model. See Anderson et al. (2004) for the original full equation.

  • 3

     The time for one production to fire in ACT-R is 50 ms. Because the current model aims at capturing the tradeoff and not the details of perceptual encoding, it is assumed that perceptual encoding of a single stimulus can be accomplished by one production (i.e., 50 ms). In the medium-cost condition, an additional 500 ms is added in account for the extra time involved in a mouse movement. In the high-cost condition, the 2-s lockout time is added in addition to the 500 ms movement time.

Ancillary