SEARCH

SEARCH BY CITATION

Keywords:

  • Multitasking;
  • Performance Operating Characteristic;
  • Cognitively Bounded Rational Analysis;
  • Cognitive modeling;
  • Performance trade-offs

Abstract

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Experiment
  5. 3. A CBRA model of tracking-while-typing
  6. 4. General discussion
  7. Acknowledgments
  8. References

We report the results of a dual-task study in which participants performed a tracking and typing task under various experimental conditions. An objective payoff function was used to provide explicit feedback on how participants should trade off performance between the tasks. Results show that participants’ dual-task interleaving strategy was sensitive to changes in the difficulty of the tracking task and resulted in differences in overall task performance. To test the hypothesis that people select strategies that maximize payoff, a Cognitively Bounded Rational Analysis model was developed. This analysis evaluated a variety of dual-task interleaving strategies to identify the optimal strategy for maximizing payoff in each condition. The model predicts that the region of optimum performance is different between experimental conditions. The correspondence between human data and the prediction of the optimal strategy is found to be remarkably high across a number of performance measures. This suggests that participants were honing their behavior to maximize payoff. Limitations are discussed.


1. Introduction

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Experiment
  5. 3. A CBRA model of tracking-while-typing
  6. 4. General discussion
  7. Acknowledgments
  8. References

Multitasking often requires trade-offs to be made in terms of how well each task is performed (e.g., task time, number of errors made). Such performance trade-offs can be described by plotting Performance Operating Characteristics, which show how the performance of separate tasks vary together systematically (Navon & Gopher, 1979; Norman & Bobrow, 1975). There is, however, a large range of strategies that might be deployed to manage task performance in a given multitasking situation (e.g., Brumby, Howes, & Salvucci, 2007). Why time is allocated differentially to each task, and why particular patterns of task interleaving are adopted, must reference the relative success of those different strategies for allocating attention between tasks (see also Payne, Duggan, & Neth, 2007). Such consideration of the strategic choices made by people in multitasking settings naturally supposes that an optimal performance trade-off might be defined. Given specific feedback about optimal performance, the question naturally becomes: Can people multitask optimally?

In line with this adaptive view, previous research has shown that people can adapt their behavior to prioritize one task over another in dual-task settings, and in this way take up different points on the Performance Operating Characteristic (e.g., Brumby, Salvucci, & Howes, 2009; Gopher, 1993; Horrey, Wickens, & Consalus, 2006; Janssen & Brumby, 2010; Levy & Pashler, 2008). However, in these studies the verbal instructions given to participants to prioritize one task over another might have been open to differences in subjective interpretation. Moreover, there is no formal method for identifying the optimal point in the performance trade-off curve. For this to be done, a quantitative payoff function must be defined against which different strategies can be evaluated.

Quantitative payoff functions have been used in experimental psychology to provide explicit instructions to participants on how the required tasks should be performed. For example, a payoff function might be used to inform participants how they should trade responding quickly to the appearance of stimuli against the risk of making a response error (e.g., Schumacher et al., 1999). Howes, Lewis, Vera, and colleagues (Howes, Lewis, & Vera, 2009; Howes, Vera, Lewis, & McCurdy, 2004; Lewis, Vera, & Howes, 2004; Vera, Howes, McCurdy, & Lewis, 2004) have taken the use of payoff functions one step further by putting forward the hypothesis that skilled human performance can be understood as a utility maximization problem that is constrained by cognitive architecture, knowledge, and experience. In other words, the idea is to assume that people are boundedly optimal. A payoff function can be used to identify this optimum solution, as was done successfully in the Psychological Refractory Period (PRP) paradigm (see Howes et al., 2009, for details).

However, in some respects the PRP task is simple: Stimuli appear at their own pace and single responses need to be made. Slightly more complex are dynamic discretionary task interleaving scenarios, where participants need to decide themselves when to switch attention from one dynamic task to another. In these scenarios, the use of payoff functions has been limited. For example, payoff functions have been used to motivate participants to perform to a certain criterion (e.g., Hornof, Zhang, & Halverson, 2010) or to demonstrate that participants use payoff as an incentive to spend more time on one task over another (e.g., Wang, Proctor, & Pick, 2007).

In this paper, we also use a payoff function in a dynamic discretionary task interleaving paradigm. However, we will follow the methodology of Howes et al. (2009) and use a payoff function to investigate whether participants adopt the optimum strategy for maximizing payoff. This is in line with the original intention of work that inspired research on Performance Operating Characteristics, Signal Detection Theory, and Receiver Operating Characteristics to identify strategies that maximize utility (Swets, Tanner, & Birdsall, 1961).

In our task environment, participants had to keep a randomly moving cursor inside a circular area and type a string of digits, but they could only see and control one task at a time. Participants’ performance was captured in a single payoff score, which reflected the payment the participant received at the end of the study.

Tracking tasks have been used in several multitasking studies (e.g., Gopher, 1993; Hornof et al., 2010; Kieras, Meyer, Ballas, & Lauber, 2000; Lallement & John, 1998; Salvucci & Taatgen, 2008). The work presented here builds on and extends this work by showing how a payoff function enables us to bind normative cognitive models with experimental observations of multitasking behavior, and specifically, to show how strategy choice in dynamic discretionary task interleaving paradigms can be better understood by comparing observed performance to a prediction of optimal performance for maximizing payoff.

2. Experiment

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Experiment
  5. 3. A CBRA model of tracking-while-typing
  6. 4. General discussion
  7. Acknowledgments
  8. References

2.1. Method

2.1.1. Participants

Eight participants (four female) between 20 and 35 years of age (= 23 years) from the subject pool at UCL participated for monetary compensation. Payment was based on performance (details are provided in the Materials section). The total payment achieved by participants ranged between £7.13 and £11.45 (= £9.14).

2.1.2. Materials

The dual-task setup required participants to perform a continuous tracking task and a discrete typing task, presented on a single 19-inch monitor with a resolution of 1280 × 1024 pixels. Fig. 1 shows the layout of the tasks on the display. The typing task was presented on the left side and the tracking task on the right. Each task was presented within a 450 × 450 pixels area, with a vertical separation of 127 pixels between the tasks.

image

Figure 1.  Position of the two tasks in the interface.

Download figure to PowerPoint

The tracking task required participants to keep a square cursor that drifted about the display in a random fashion inside a target circle (see Fig. 1). The cursor was 10 × 10 pixels and the target had a radius of either 80 (small target) or 120 pixels (large target). A random walk function was used to vary the position of the cursor in the display. The rate at which the cursor drifted about the display was varied between different experimental conditions. In a low noise condition, the random walk had a mean of zero and standard deviation of three pixels per update, while in a high noise condition the random walk had a mean of zero and standard deviation of five pixels per update. Updates occurred approximately once every 25 ms. To control the position of the cursor in the tracking display, participants used a Logitech Extreme 3D Pro joystick with their right hand. The drift function of the cursor was suspended whenever the joystick angle was greater than ±0.08 (the maximum angle was ±1). The speed at which the cursor could be moved was scaled by the angle, with a maximum of five pixels per 25 ms.

The typing task required participants to enter a string of 20 digits using a numeric keypad with their left hand. The string was made up of the digits 1–3, where each digit occurred at least six times in a given sequence. Digits were presented in a random order with the constraint that no single digit was presented more than three times in a row in the sequence (e.g., “31213321231322231123” as in Fig. 1). When a digit was entered correctly it was removed from the to-be-entered sequence. In this way, the left-most digit on the display was always the next one to be entered. When an incorrect digit was typed, the string would not progress. No additional signal was given to indicate this error.

The study used a forced interleaving paradigm, in which only one of the two tasks was visible and could be worked on at any moment in time. By default the typing task was visible and the tracking task was covered by a gray square. Holding down the trigger of the joystick made the tracking task visible and covered the typing task. Releasing the trigger covered the tracking task and made the typing task visible once more. Input was only for the visible task and any input for the covered task was ignored (recall that the tracking task only received input from the joystick while the typing task only received input from the keyboard).

2.1.3. Design

The study manipulated aspects of the tracking task using a 2 (cursor noise: low vs. high) × 2 (target size: small vs. large) within-subjects design. The main dependent variables were the time required to complete the typing task, the maximum distance of the cursor from the center of the target circle, and the total time the cursor was outside of the target circle.

Participants were remunerated based on their performance using an objective payoff function. The payoff function was designed to encourage fast completion of the typing task while also encouraging participants to keep the cursor inside the target. The payoff (in pounds) received following a given trial was defined as:

  • image(1)

The minimum payoff for a given trial was limited to −0.20 pounds (i.e., a loss). The gain component was based on the total time required to complete a dual-task trial (in seconds):

  • image(2)

This function was determined using pilot studies, to make sure participants mostly gained money. To encourage participants to keep the cursor inside the target circle of the tracking task, a tracking penalty was applied in trials where the cursor crossed the target boundary:

  • image(3)

With this penalty, £0.10 was lost when the cursor was outside of the radius for 0.625 s, and £0.20 was lost when it was outside of the radius for 1.25 s.

To encourage accurate typing, a digit penalty deduced £0.01 from the total payoff whenever an incorrect digit was entered. In the remainder of this paper, we will not look at the effect of digit penalty on payoff, as the total number of errors was relatively low, and in most trials no errors were made (see Results). We leave a further investigation of errors to future work and refer the interested reader to Smith, Lewis, Howes, Chu, and Green (2008) for a model that investigates the impact of errors on performance.

2.1.4. Procedure

Participants were informed that they would be required to perform a series of dual-task trials and that they would be paid based on their performance. A participant’s payment was based on the cumulative payoff over the course of the study, in addition to a base payment of £3. Participants were told that they would gain more points by completing the typing task as quickly as possible, but that they would lose points if they made a typing error or if the cursor drifted outside of the target area in the tracking task. We chose not to give participants a formal description of the payoff function, but instead provided explicit feedback after every dual-task trial with the payoff score achieved.

After explaining how to perform each of the tasks participants performed two single-task training trials for each task and two dual-task practice trials. Participants were instructed that for dual-task trials only one of the two tasks would be visible and controllable at any moment in time, and they were instructed how to switch between tasks using the trigger button on the joystick.

Participants then completed four blocks of experimental trials (one for each experimental condition). In the first two blocks, participants experienced a single noise level, either low or high noise. The noise level was randomly assigned to participants and balanced across participants. On the first block a radius size (small or large) was also randomly assigned, and on the second block the other radius level was assigned. For the third and fourth block this order was repeated, but with another level for noise. For each block, participants completed five single-task tracking trials, five single-task typing trials, and twenty dual-task trials. The dual-task trials were further grouped into sets of five trials, with a short pause between each set. The total procedure took about 1 h to complete.

2.2. Results

Across all keystrokes in single-task typing trials, participants on average typed 2.5% (range 0.5–5.2%) of their keystrokes incorrectly (81 out of 3,281 keystrokes). At the trial level, 0, 1, 2, or more errors were made on, respectively, 61.9%, 29.4%, 5%, and 3.8% of the trials.

In the dual-task trials, the number of typing trials was also low. Participants on average typed 3.6% (range 1.2–5.4%) of their keystrokes incorrect (481 out of 13,281 keystrokes). At the trial level, 0, 1, 2, or more errors were made on, respectively, 52.5%, 29.4%, 12.0%, and 6.1% of the trials.

We regard the occurrence of errors interesting, but their occurrence is too low to draw any conclusions from (i.e., on most trials there are no errors). We therefore do not look at the effect that errors had on performance and leave this for future work.

Also, we focus on performance during the last five dual-task trials of each experimental condition, as these reflect a period during which the participant had had time to adapt their behavior to the payoff function based on the feedback received. A 2 (cursor noise) × 2 (target size) analysis of variance (anova) was used for all statistical analysis with a significance level of .05.

2.2.1. Overall performance

We first consider the effect of varying aspects of the tracking task on the time required to complete the typing task, the maximum distance of the cursor from the center of the target circle in the tracking task, and the mean time the cursor was outside the target area. It was found that trial time was significantly longer when there was greater noise in the tracking task (= 11.17 s, SD = 4.32 s) than when there was a lower level of noise in the tracking task (= 7.51 s, SD = 2.00 s), F(1, 7) = 15.07, < .01. Trials were also longer when the target in the tracking task was smaller (= 10.59 s, SD = 4.01 s) than when it was larger (= 8.09 s, SD = 3.22 s), F(1, 7) = 11.84, = .01. There was no significant interaction, F(1, 7) < 1.

In the tracking task, we consider the maximum distance of the cursor from the center of the target over the course of a trial. It was found that the cursor drifted more when there was a higher level of noise (= 95 pixels, SD = 15 pixels) than when there was a lower level of noise (= 61 pixels, SD = 8 pixels), F(1, 7) = 33.42, < .001. There was no effect of target size on the maximum distance that the cursor drifted over a trial, F(1, 7) = 1.19, = .31, nor was the interaction effect significant, F(1, 7) < 1.

Another measure of performance in the tracking task is the average time the cursor was outside of the target area per trial. Participants let the cursor remain outside of the target area for longer when there was high noise (= 0.36 s, SD = 0.45 s), compared to when there was low noise (= 0.04 s, SD = 0.10 s), F(1, 7) = 7.28, = .03. The cursor also spent more time outside of the target area when the target area was small (= 0.34 s, SD = 0.05 s), compared to when it was large (= 0.05 s, SD = 0.11 s), F(1, 7) = 13.26, < .01. The interaction was not significant, but there was evidence of a trend, F(1, 7) = 4.58, = .07. This trend reflects that in the low noise, large target condition the cursor never crossed the target area, whereas in the high noise, small target condition the cursor crossed the target area for over half a second.

These differences in overall task performance between conditions are somewhat expected and unsurprising because they partly reflect differences in the difficulty of the tracking task. We were far more interested in how this performance was achieved. We next consider the dual-task interleaving strategy that was adopted in each condition.

2.2.2. Strategies

Two aspects determine a strategy: (a) the number of digits typed during each visit to the typing window and (b) the amount of time spent in the tracking window per visit to this window. Fig. 2 shows these two basic strategy dimensions for each of the four conditions. For the number of digits typed we only considered correct digits. It can be seen that for each experimental condition there is a unique point in this strategy space—strategies differ between conditions. The number of digits entered per visit increased with an increase in target size, F(1, 7) = 17.4, < .01, and it also increased with a decrease in cursor noise. That is, more digits were typed when it took longer for the cursor to cross the boundary, F(1, 7) = 15.18, < .01. There was no significant interaction, F(1, 7) = 3.24, = .12.

image

Figure 2.  Plot of the mean number of digits typed and time spent tracking, both per visit. Error bars depict standard errors.

Download figure to PowerPoint

It can also be seen in Fig. 2 that the time spent in the tracking window per visit increased with an increase in the noise associated with the cursor’s movement, F(1, 7) = 14.98, = .01. An interaction effect was present as visit time was particularly short in the low noise, large radius condition, F(1, 7) = 11.55, = .01. There was no significant effect of radius, F(1, 7) < 1.

3. A CBRA model of tracking-while-typing

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Experiment
  5. 3. A CBRA model of tracking-while-typing
  6. 4. General discussion
  7. Acknowledgments
  8. References

The results show that participants adapted their dual-task behavior to changes in the difficulty of the tracking task by varying the amount of time that was given to each task before switching to the other task. However, what these results do not show is whether participants were adopting a strategy that is optimal in terms of maximizing the expected payoff that could be achieved in each condition, both for the individual task (tracking and typing) and the combination of tasks. To answer this question we developed a Cognitively Bounded Rational Analysis model (Howes et al., 2009) of aggregate human performance. This framework is particularly useful for comparing the performance of alternative strategies, allowing strategies to be discriminated based on the payoff achieved. The model developed here is inspired by our previous work in developing models of a dialing-while-driving dual-task setup (e.g., Brumby, Salvucci, & Howes, 2007; Brumby et al., 2009; Janssen & Brumby, 2010). Both dual-task environments share core characteristics, but the current work differs in that it incorporates an explicit payoff function against which various dual-task interleaving strategies can be evaluated. In the next section, we describe the model that was used to determine whether people were selecting strategies that would maximize the financial payout that could be achieved in each condition.

3.1. Model development

3.1.1. Tracking model

What did people do when visiting the tracking window? The crucial question for developing a model of the tracking task is to understand how people set the angle of the joystick based on the position of the cursor in the display. Fig. 3 shows the mean values for discrete bins of five pixels for the horizontal axes (vertical data are similar). We fitted a linear function (shown as a dotted line):

  • image(4)
image

Figure 3.  Plot of the angle of the joystick as a function of distance from the target. The dashed line shows a fitted function.

Download figure to PowerPoint

The joystick had a maximum angle of (−)1. This shows that participants’ behavior in the tracking task can be captured by a simple linear function that sets the angle of the joystick based on the position of the target within the display. To implement this model, as in the experiment, the speed of the cursor is calculated by multiplying the angle of the joystick with a value of five pixels. Speed is calculated once every 250 ms of tracking, and the cursor position is updated every 25 ms based on this speed value. As in the experiment, the cursor can only be controlled when the tracking window is open. The total time spent tracking in dual-task is varied as part of the strategy (see below).

3.1.2. Typing model

To model the typing task we fitted model performance to human single-task typing performance data. To get a measure of how long it took participants to enter a digit in the typing task, we take the mean single-task inter-keypress interval, which was 260 ms. This value was calculated by taking the mean value of participants’ total typing time and dividing this by the number of to-be-entered digits (20). In this way, errors were taken only indirectly into account. We use this time estimate to model the time it takes to enter a single digit in the typing task.

3.1.3. Dual-task model

The dual-task model works as followed. The model starts off with typing a series of digits (the length of which is varied as a strategy). The time to type each digit is taken from our single-task model (260 ms). For switching between typing and tracking a switch cost of 250 ms is incurred, based on experimental data (time between last key press and pressing the trigger on the joystick: 247 ms). The model then tracks the cursor for a designated amount of time (varied between runs as a strategy aspect). When it switches back to typing, a switch cost of 180 ms is incurred (time between releasing the trigger and pressing the first key, corrected for the single task typing time: 185 ms).

3.1.4. Strategies

We used the basic model described above to explore performance of a variety of different dual-task strategies. A strategy is determined by the number of digits that are typed in sequence during a visit to the target window. We consider only a subset of 20 simple strategies that placed a consistent number of digits during each visit to the typing task (between 1 and 20 digits), with the exception of the last visit during which the remaining digits were placed (e.g., strategy 6-track-6-track-6-track-2). In addition, for each visit to the tracking task, more or less time can be spent on tracking. We systematically explored performance for models that spent between 250 and 3,000 ms on tracking during each visit to the tracking window, using steps of 250 ms (i.e., 12 alternatives). This gave a total of 229 (20 × 12–11) strategy alternatives (see also, Brumby, Salvucci, et al., 2007; Brumby et al., 2009; Janssen & Brumby, 2010, for a similar approach to modeling driver distraction).

The objective function for rating performance is similar as in the experiment (see Eqs. 1-3) with the exception that the model does not make typing errors. For each strategy alternative 100 runs were performed. Mean performance is reported.

3.2. Model results

The first question of interest was whether the model would fit the experimental data. To do this, we hardcoded a strategy for each condition that typed the same number of correct digits per visit and spent about the same amount of time tracking as participants did. We took these values within one standard error of the human means, as our model’s strategy alternatives were more discrete than the human data (e.g., the model’s tracking time was explored in discrete steps of 250 ms). With these values set, we asked whether the model’s performance fitted the total trial time, maximum deviation, and time outside the target area in each experimental condition observed in the human data. This is important so as to know that we have a reasonable calibration of the model’s performance relative to the human data. Model performance was within two standard errors of the human data for these variables.

Given that we can be confident that the model is reasonably calibrated to the observed strategies, we can now use the model to evaluate the payoff achieved by different (unobserved) dual-task interleaving strategies. Fig. 4 shows a plot of the maximum number of digits typed per visit to the typing window versus payoff. In this figure (and Figs. 5 and 6), the performance predictions of the model for each strategy alternative are represented by colored circles. The color of the circle reflects the average payoff that the model gained over 100 simulations when this strategy was applied. The warmer the color, the higher the score (i.e., the higher on the vertical axis in Fig. 4). The maximum score is £0.16, and each change in color reflects a change in payoff of £0.02.

image

Figure 4.  Plot of the mean number of digits typed per visit to the typing window versus predicted payoff per trial for the modeled strategies per condition. Color represents the average payoff achieved by the model using that strategy. Human results are shown as black points with standard error.

Download figure to PowerPoint

image

Figure 5.  POCs of trial time versus maximum deviation for the modeled strategy alternatives per condition. Color represents the average payoff achieved by the model using that strategy. Human results are shown as black points with standard error. The dashed line shows the target boundary.

Download figure to PowerPoint

image

Figure 6.  POCs of trial time versus time the cursor was outside of the target area for the modeled strategy alternatives per condition. Color represents the average payoff achieved by the model using that strategy. Human results are shown as black points with standard error.

Download figure to PowerPoint

The horizontal axis of Fig. 4 shows the maximum number of digits that the model typed per visit to the target window. Each of our 20 simple strategies takes a unique value on this axis (e.g., strategy 6-track-6-track-6-track-2 is plotted at value 6 on the horizontal axis). For each of these simple strategies, we explored multiple strategy alternatives based on how much time the model spent on tracking. This resulted in multiple points for each value on the horizontal axis. In three out of the four conditions, the human data (black point, with standard error bars) lie within the region of the highest payoff. That is, on average participants typed the optimum number of digits per visit to the target window so as to achieve the highest payoff. Note that in the small radius, large noise condition participants did not achieve the highest score—they should have typed less digits per visit to the typing window.

The analysis above suggests that participants selected appropriate strategies in each condition. To investigate whether this strategy also resulted in good overall performance, we plotted Performance Operating Characteristics (POCs). Recall that POCs display performance on one task against performance on the other task (Navon & Gopher, 1979; Norman & Bobrow, 1975). We included two types of POCs. In Fig. 5, the POCs are plotted for the total trial time and the maximum deviation of the cursor from the center of the target. In Fig. 6, the POCs are plotted for the total trial time and the total time the cursor spent outside of the target area per trial. Again the color of the model data represents the average payoff achieved using this strategy.

We highlight some general observations of these POCs:

  • 1
     For each condition the shape of the POC differs.
  • 2
     The scores that can be achieved differ between conditions, as indicated by different color ranges in each condition.
  • 3
     The best performing strategies (i.e., the regions with the warmest colors) tend to cluster on the outer edge (left side and bottom side) of the strategy space: the trade-off curve. That is, the best strategies make an optimal trade-off for performance on the combination of the two tasks.
  • 4
     In addition to point 3, for the current payoff function the optimum region is at different sections of the trade-off curve for some of the conditions. The biggest contrast is in Fig. 5 between the low noise, large target condition and the high noise, small target condition. In the former, the best score is achieved by letting the cursor drift completely (i.e., the best performance is at the top left), whereas in the latter condition the optimum is at the inflection point (i.e., the middle of the curve, on the outside, where it crosses the dashed line). The model is essential for this assessment, as traditional POCs cannot predict optimal regions by themselves. Inherently, the exact location of the optimum region can also shift with a change in the payoff function.

Leaning on this fourth point, our analysis helps to bracket optimal performance. For each of the measures of total trial time, maximum deviation of the cursor (see Fig. 5), and time spent outside of the target area (see Fig. 6), the model predicts that the optimal region lies in a different range of values. This is consistent with our finding in the human data for these measures that showed main effects of radius, or noise, or significant interactions of these factors. Note that this way of bracketing differs from bracketing methodologies that identify the fastest and slowest strategies for performance based on performance time (e.g., Kieras & Meyer, 2000). We can bracket performance for the best strategy alternatives (and others if necessary), based on the predicted payoff of those strategies. Similar to, for example, the work by Kieras and Meyer (2000), performance of these strategies can then be expressed in multiple dimensions of performance (e.g., in our case trial time, maximum deviation of the cursor, and time spent outside of the target area).

Fig. 5 suggests that for the different performance measures, human performance is around or at the optimum. In all four conditions, human performance overlaps with the optimum range of values for total trial time and maximum deviation of the cursor from the center of the target (see Fig. 5). Fig. 6 suggests that for three conditions, human performance also overlaps with the optimum range of values for the total time the cursor was outside of the target area. It does not overlap in the high noise, small radius condition (see the bottom left graph in Fig. 6).

To close, the correspondence between performance predictions of the best model strategy alternatives and human data can also be assessed using mean performance. Fig. 7 shows bar plots of mean human performance and mean model performance for three measures of performance. Model data are the mean performance for strategy alternatives that fall in the highest scoring region (i.e., in the region with the warmest color in Figs. 4–6). The correspondence between the human and model bars is surprisingly high—error bars between model and human data overlap in most instances. This also reflects in R2 and root mean squared error (RMSE) values. The fit values are as follows for trial time (R= 0.88, RMSE = 1.63 s), maximum deviation (R= 0.95, RMSE = 5.27 pixels), and time spent outside of the target area (R= 0.98, RMSE = 0.21 s). These values can be considered to be high, given that the model was not fitted to optimize R2 or RMSE values. Rather, the prediction of the model was based on the set of strategy alternatives that would achieve the highest payoff in each condition.

image

Figure 7.  Bar plot of human and model performance in each condition. Model performance is the mean performance for strategy alternatives that fall in the highest scoring region (i.e., in the region with the warmest color in Figs. 4–6). Error bars show standard error. The three plots show (A) total trial time (in seconds), (B) maximum deviation of the cursor from the center of the target (in pixels), and (C) total time the cursor was outside of the target area (in seconds).

Download figure to PowerPoint

4. General discussion

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Experiment
  5. 3. A CBRA model of tracking-while-typing
  6. 4. General discussion
  7. Acknowledgments
  8. References

In this paper, we have presented an experiment and a model of a tracking-while-typing dual-task setup. A good feature of the task environment, in which participants need to track a cursor and type in digits, is that it translates performance on both tasks into a single performance score. This allowed us to move beyond observations that participants trade off performance in tasks, as done in classical dual-task research (Navon & Gopher, 1979; Norman & Bobrow, 1975) and for example in research on dual-task driving behavior (e.g., Janssen & Brumby, 2010). Following Howes et al. (2009), we were able to bracket optimum performance and to demonstrate situations where participants made performance trade-offs in an optimal manner, so as to maximize payoff. This is in line with the original objectives of work that inspired work on Performance Operating Characteristics (Navon & Gopher, 1979; Norman & Bobrow, 1975)—work on Signal Detection Theory and Receiver Operating Characteristics (Swets et al., 1961).

The goal of this paper is not to argue that objective functions are the most prevalent aspect of performance in the real world. However, they make it possible to quantify how good performance is. This contrasts with previous work on discretionary task interleaving where verbal instructions on how to trade performance on each task is given (e.g., Brumby et al., 2009; Gopher, 1993; Horrey et al., 2006; Janssen & Brumby, 2010; Levy & Pashler, 2008), where payoff functions were used to motivate participants to perform to a certain criterion (e.g., Hornof et al., 2010), or where payoff functions were used to demonstrate that performance is sensitive to a change in payment (e.g., Wang et al., 2007). In contrast, we can define optimal performance on the combined tasks in terms of maximizing payoff.

In our task environment, participants selected strategies that had the potential to achieve the maximum payoff in three out of four conditions (see Fig. 4). In most conditions, participants optimized each of the individual performance measures (total trial time, maximum deviation of the cursor from center, and time spent outside of the target area). Their performance overlaps with the bracketed optimum performance of the model (see Figs. 5 and 6), and the measures of fit between mean human and mean model performance of the best scoring strategy alternatives are high (see Fig. 7).

In the low noise condition, participants also made the optimal trade-off on overall combined task performance, as their performance overlaps with the model region of maximum payoff (see Fig. 5). In the high noise condition, human performance seems not to overlap with the region of best combined performance. This assessment is dependent on the cutoff point for the best region. Given our cutoff point, one explanation why performance might not lie in the optimum region in this condition is that it is hard for participants to assess how well they performed on the tracking task; this task is covered up part of the time. Participants can get a sense of the maximum deviation when they open the tracking window, but they cannot tell how long the cursor has been outside the target area.

In addition, the model performance is based on average performance over 100 samples (or trials) per strategy, whereas human performance is based on performance over only five trials for each participant. As the position of the cursor in both the model and the experiment is manipulated by a noise function, it might be that the sample of noise values that the human participants experienced differs from the averaged sample that the model received (see also Fig. 7C—the variance in the human data is very high in the high noise, small target condition). This is of particular influence in the high noise condition, as the extremes are further apart (due to the higher noise value). One way of working around this problem is by having the model experience the same drift values that the participants experienced. Alternatively, the variance of performance could be taken into account in the assessment of the payoff of a strategy alternative.

The model was developed with a minimal set of assumptions. This was already enough to demonstrate that people can adapt performance to an objective function in some situations. Further research can investigate how people adapt their behavior to different payoff functions, which, for instance, give greater weight to performance on one of the two tasks. Experimentally it might be investigated whether there are any strategy transfer effects between the different payoff conditions. We tried to minimize this in our analysis, by only looking at data after participants experienced a condition for 15 trials. However, approaches such as the soft constraints hypothesis predict that strategy transfer effects might still remain (Gray, Sims, Fu, & Schoelles, 2006).

The model of the typing task might be refined for example to predict the effect of the different times needed to type repeating digits versus nonrepeating digits (cf. Janssen, Brumby, & Garnett, 2010). The model also does not yet give an account of typing errors, whereas typing errors do influence the payoff that is achieved by participants. More experimental data would be required to identify the nature of these errors, for example, whether they are related to speed-accuracy trade-offs (cf. Wobbrock, Cutrell, Harada, & MacKenzie, 2008). Our assumption not to include typing errors in the model is a similar assumption as was made in our dialing-while-driving models (e.g., Brumby, Salvucci, et al., 2007; Brumby et al., 2009; Janssen & Brumby, 2010), but it differs from other Cognitively Bounded Rational Analysis models (e.g., Smith et al., 2008).

At a different level of analysis, the role of eye movements can be considered to explore a wider variety of strategies (cf. Hornof et al., 2010), such as strategies in which some visits to the typing task window are only spent on studying, and not typing digits.

Finally, our model gives an account of participants’ ability to interleave the two tasks of tracking and typing in an optimal way. However, the model does not explain how participants learn to make this optimal trade-off. In order to do this, theories of learning need to be incorporated (e.g., Erev & Gopher, 1999).

Acknowledgments

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Experiment
  5. 3. A CBRA model of tracking-while-typing
  6. 4. General discussion
  7. Acknowledgments
  8. References

This work was supported by EPSRC grant EP/G043507/1. The paper is an extension of a paper we presented at the 10th International Conference on Cognitive Modeling (Janssen, Brumby, Dowell, & Chater, 2010). We thank Julian Marewski and two anonymous reviewers for their valuable comments on that paper. A preliminary version of this work was also presented at the ICCM Doctoral Consortium. We thank attendees for their comments, in particular John Laird for his suggestion to let the model experience the exact noise sample that the participants experienced rather than values of a noise function. We thank Richard Young for the suggestion to use colored pictures to display payoff. Finally, we thank Wayne Gray for his review of this paper.

References

  1. Top of page
  2. Abstract
  3. 1. Introduction
  4. 2. Experiment
  5. 3. A CBRA model of tracking-while-typing
  6. 4. General discussion
  7. Acknowledgments
  8. References
  • Brumby, D. P., Howes, A., & Salvucci, D. D. (2007). A cognitive constraint model of dual-task trade-offs in a highly dynamic driving task. In B.Beqole, S.Payne, E.Churchill, R.St. Armant, D.Gilmore, & M. B.Rosson (Eds.), Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 233242). New York: ACM Press.
  • Brumby, D. P., Salvucci, D. D., & Howes, A. (2007). Dialing while driving? A bounded rational analysis of concurrent multi-task behavior. In R. L.Lewis, T. A.Polk, & J. E.Laird (Eds.), Proceedings of the 8th International Conference on Cognitive Modeling (pp. 121126). Ann Arbor, MI: University of Michigan.
  • Brumby, D. P., Salvucci, D. D., & Howes, A. (2009). Focus on driving: How cognitive constraints shape the adaptation of strategy when dialing while driving. In S.Greenberg, S. E.Hudson, K.Hinkley, M. R.Morris, & D. R.OlsenJr (Eds.), Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 16291638). New York: ACM Press.
  • Erev, I., & Gopher, D. (1999). A cognitive game-theoretic analysis of attention strategies, ability, and incentives. In D.Gopher & A.Koriat (Eds.), Attention and performance XVII: Cognitive regulation of performance: Interaction of theory and application (pp. 343371). Cambridge, MA: MIT Press.
  • Gopher, D. (1993). The skill of attention control: Acquisition and execution of attention strategies. In D. E.Meyer & S.Kornblum (Eds.), Attention and performance XIV: Synergies in experimental psychology, artificial intelligence, and cognitive neuroscience (pp. 299322). Cambridge, MA: MIT Press.
  • Gray, W. D., Sims, C. R., Fu, W. T., & Schoelles, M. J. (2006). The soft constraints hypothesis: A rational analysis approach to resource allocation for interactive behavior. Psychological Review, 113, 461482.
  • Hornof, A. J., Zhang, Y., & Halverson, T. (2010). Knowing where and when to look in a time-critical multimodel dual task. In S. E.Hudson, G.Fitzpatrick, W. K.Edwards, T.Rodden, & E.Mynatt (Eds.), Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 21032112). New York: ACM Press.
  • Horrey, W. J., Wickens, C. D., & Consalus, K. P. (2006). Modeling drivers’ visual attention allocation while interacting with in-vehicle technologies. Journal of Experimental Psychology: Applied, 12, 6778.
  • Howes, A., Lewis, R. L., & Vera, A. (2009). Rational adaptation under task and processing constraints: Implications for testing theories of cognition and action. Psychological Review, 116, 717751.
  • Howes, A., Vera, A., Lewis, R. L., & McCurdy, M. (2004). Cognitive constraint modeling: A formal approach to supporting reasoning about behavior. In K.Forbus, D.Gentner, & T.Regier (Eds.), Proceedings of the 26th Annual Meeting of the Cognitive Science Society (pp. 595600). Mahwah, NJ: Lawrence Erlbaum Associates.
  • Janssen, C. P., & Brumby, D. P. (2010). Strategic adaptation to performance objectives in a dual-task setting. Cognitive Science, 34, 15481560.
  • Janssen, C. P., Brumby, D. P., Dowell, J., & Chater, N. (2010). A cognitively bounded rational analysis model of dual-task performance trade-offs. In D. D.Salvucci & G.Gunzelmann (Eds.), Proceedings of the 10th International Conference on Cognitive Modeling (pp. 103108). Philadelphia, PA: Drexel University.
  • Janssen, C. P., Brumby, D. P., & Garnett, R. (2010). Natural break points: Utilizing motor cues when multitasking. In Proceedings of the 54th Annual Meeting of the Human Factors and Ergonomics Society (pp. 482486). San Francisco, CA: Human Factors and Ergonomics Society.
  • Kieras, D. E., & Meyer, D. E. (2000). The role of cognitive task analysis in the application of predictive models of human performance. In J. M.Schraagen, S. F.Chipman, & V. L.Shalin (Eds.), Cognitive task analysis (pp. 237260). Mahwah, NJ: Erlbaum.
  • Kieras, D. E., Meyer, D. E., Ballas, J. A., & Lauber, E. J. (2000). Modern computational perspectives on executive mental processes and cognitive control: Where to from here? In S.Monsell & J.Driver (Eds.), Attention and performance XVIII: Control of cognitive processes (pp. 681712). Cambridge, MA: MIT Press.
  • Lallement, Y., & John, B. (1998). Cognitive architecture and modeling idiom: An examination of three models of the wickens task. In M. A.Gernsbacher & S. J.Derry (Eds.), Proceedings of the Twentieth Annual Conference of the Cognitive Science Society (pp. 597602). Mahwah, NJ: Lawrence Erlbaum Associates.
  • Levy, J., & Pashler, H. (2008). Task prioritisation in multitasking during driving: Opportunity to abort a concurrent task does not insulate braking responses from dual-task slowing. Applied Cognitive Psychology, 22, 507525.
  • Lewis, R., Vera, A., & Howes, A. (2004). A constraint-based approach to understanding the composition of skill. In M.Lovett, C.Schunn, C.Lebiere, & P.Munro (Eds.), Proceedings of the Sixth International Conference on Cognitive Modeling (pp. 148153). Mahwah, NJ: Lawrence Erlbaum.
  • Navon, D., & Gopher, D. (1979). On the economy of the human-processing system. Psychological Review, 86, 214255.
  • Norman, D. A., & Bobrow, D. G. (1975). On data-limited and resource-limited processes. Cognitive Psychology, 7, 4464.
  • Payne, S. J., Duggan, G. B., & Neth, H. (2007). Discretionary task interleaving: Heuristics for time allocation in cognitive foraging. Journal of Experimental Psychology: General, 136, 370388.
  • Salvucci, D. D., & Taatgen, N. A. (2008). Threaded cognition: An integrated theory of concurrent multitasking. Psychological Review, 115, 101130.
  • Schumacher, E., Lauber, E., Glass, J., Zurbriggen, E., Gmeindl, L., Kieras, D. E., & Meyer, D. E (1999). Concurrent response-selection processes in dual-task performance: Evidence for adaptive executive control of task scheduling. Journal of Experimental Psychology: Human Perception and Performance, 25, 791814.
  • Smith, M. R., Lewis, R. L., Howes, A., Chu, A., & Green, C. (2008). More than 8,192 ways to skin a cat: Modeling behavior in multidimensional strategy spaces. In B. C.Love, K.McRae, & V. M.Sloutsky (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 14411446). Austin, TX: Cognitive Science Society.
  • Swets, J. A., Tanner, W., & Birdsall, T. (1961). Decision processes in perception. Psychological Review, 68, 301340.
  • Vera, A., Howes, A., McCurdy, M., & Lewis, R. (2004). A constraint satisfaction approach to predicting skilled interactive cognition. In E.Dykstra-Erickson & M.Tscheligi (Eds.), Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 121128). New York: ACM Press.
  • Wang, D. D., Proctor, R. W., & Pick, D. F. (2007). Acquisition and transfer of attention allocation strategies in a multiple-task work environment. Human Factors, 49, 9951004.
  • Wobbrock, J., Cutrell, E., Harada, S., & MacKenzie, I. (2008). An error model for pointing based on fitts’ law. In M.Burnett, M. F.Costabile, T.Catarci, B.De Ruyter, D.Tan, M.Czerwinski, & A.Lund (Eds.), Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 16131622). New York: ACM Press.