This section describes the data examined in this paper, which were collected in 2006 as part of a nationwide US survey. In addition to the questions described in this section, the survey also included questions on respondents' sources, uses, and values of weather forecasts; perceptions and interpretations of weather forecasts, including uncertainty information; sociodemographic characteristics; and other topics (see Morss et al., 2008; Lazo et al., 2009). Several analyses were performed in conjunction with data from these other questions, in order to examine possible influences on use of forecast information.
2.1. Survey development, implementation and respondent population
The survey instrument was developed and pretested using standard principles for developing survey questions and conducting survey research (Schuman and Presser, 1996; Dillman, 2000; Tourangeau et al., 2000). The survey was implemented in November 2006 on the Internet with a sample provided by a survey sampling company, designed to be representative of the US population reachable online. The respondent population includes people from every US state and the District of Columbia. Its sociodemographic characteristics are generally similar to that of the US population (U.S. Census Bureau, 2006), except that it is somewhat older and more educated and under represents people with very low and high incomes. While the respondent population is not a random sample of the general US population, it is more diverse and representative than previous related work with convenience samples or students. For further detail on the survey development, implementation and respondent population, see Morss et al. (2008).
Although 1520 completed surveys were received, 55 respondents said that they did not ever use weather forecasts and were not asked most of the remaining questions, including those examined here. Thus, the analysis presented begins with data from 1465 respondents.
2.2. Threshold decision questions
The threshold decision questions asked respondents their probabilistic forecast threshold for taking protective action in two scenarios, referred to as the picnic and garden scenarios. The picnic scenario involves protection against precipitation, and the garden scenario involves protection against low temperature. These questions were motivated by a related question asked in Gigerenzer et al. (2005). Each respondent was randomly assigned to receive one of the two scenarios and the associated threshold question.
In the picnic scenario, respondents were told to suppose they have an outdoor picnic planned for tomorrow. They were then asked: ‘At what forecast chance of rain for tomorrow would you decide today to move your picnic indoors?’ There were 11 response options: forecast chance of rain from 10 to 100%, in intervals of 10%, or not moving the picnic indoors (i.e. take no action).
In the garden scenario, respondents were told to suppose they have a garden with plants that will die if the temperature drops below freezing (32 °F). They were then asked: ‘At what forecast chance that the temperature will be below freezing (32 °F) tonight would you decide today to cover your plants?’ There were 11 response options: forecast chance of temperature below freezing (32 °F) from 10 to 100%, in intervals of 10%, or not covering the plants (i.e. take no action).
The authors hypothesized that respondents would select a range of thresholds but no other specific hypotheses about the responses.
2.3. Binary decision questions
The binary decision questions asked respondents whether or not they would take protective action given different forecast information in two scenarios, referred to as the reservoir and fruit scenarios. The two scenarios are summarized in Table I and presented in further detail below. The scenarios were patterned after the cost-loss decision situation (Thompson, 1952; Thompson and Brier, 1955), in the sense that they include two decision alternatives (protective action at a cost, or no protective action) and two possible outcomes involving monetary losses ($ 100 000 damage, or no damage). For each scenario, two cost conditions ($ 10 000 or $ 20 000) for protective action were tested. The questions were developed to compare respondents' use of different types of forecast information and to examine their decisions from a cost-loss perspective.
Table I. Overview of the reservoir and fruit scenarios used in the binary decision questions described in Section 2.3 (each with two cost conditions)
| ||Reservoir scenario||Fruit scenario|
|Potential damage (monetary loss)||$ 100 000||$ 100 000|
|Damage||4 in. or more||Low temperature|
|threshold||of rain||below 32 °F|
|Cost of taking protective action||$ 10 000||$ 20 000||$ 10 000||$ 20 000|
|Number of respondents after/before removing all-yes and all-no responses (total 1233/1465)||307/362||300/367||315/363||311/373|
The reservoir scenario involves protection against precipitation and the fruit scenario involves protection against low temperature, similar to the picnic and garden scenarios, respectively, discussed for the threshold question in Section 2.2. Each respondent was (randomly) assigned to receive either precipitation or temperature scenarios for both sets of questions. In other words, respondents who received the picnic (garden) scenario for the threshold decision question received the reservoir (fruit) scenario for the binary decision questions. Each respondent was also randomly assigned to one of the two protective-action cost conditions. Thus, for the binary decision questions, the respondent population was divided into four groups (Table I).
In the reservoir scenario, respondents were told:
‘Suppose you are a manager of a local water reservoir. If there are 4 inches or more of rain tomorrow, your reservoir will overflow and flood the town, causing $ 100 000 in damages (but no injuries or deaths) that your company must pay for. You can prevent a potential flood by releasing water from your reservoir today, but releasing water will cost your company ($ 10 000 or $ 20 000).’
The four possible combinations of decisions (action or no action) and outcomes (damage or no damage) were explained. Respondents were then presented nine forecast conditions, one at a time, in random order. For each forecast condition, they were asked ‘Would you spend the ($ 10 000 or $ 20 000) to release water from your reservoir?’
In the fruit scenario, respondents were told:
‘Suppose you are a fruit grower and your crop is nearly ripe. If the temperature drops below freezing (32 °F) tonight and your crop is unprotected, it will be damaged and you will lose $ 100 000. You can prevent potential freeze damage by protecting your crop today, but protecting your crop will cost you ($ 10 000 or $ 20 000).’
As in the reservoir scenario, the four combinations of decisions and outcomes were explained. Respondents were then asked ‘Would you spend the ($ 10 000 or $ 20 000) to protect your crop?’ for each of nine forecast conditions.
The response options were ‘yes’ or ‘no’ for each question. Respondents were not allowed to return to previous questions, nor were they given information about which outcome occurred.
The nine forecast conditions tested in the reservoir and fruit scenarios are shown in Table II. The forecast conditions were designed to manipulate a few relatively simple dimensions of information presentation, using text-based formats. The ‘single-value’ forecast conditions provide deterministic forecast information, similar to that currently available in most forecasts (NRC, 2006); the forecast values are different distances from the damage threshold. Two forms of uncertainty communication were tested: ‘range’ forecasts and ‘percentage-chance’ forecasts. The range forecast conditions were designed to represent a fairly simple form of uncertainty communication, one that does not involve probabilities. The range forecasts are symmetrical about the third single-value forecast, with the first range meeting the damage threshold and the second range exceeding the threshold. The percentage-chance forecast conditions were designed to represent a somewhat more complex form of uncertainty communication; they present different probabilities of reaching or exceeding the damage threshold.
Table II. Forecast conditions presented in the binary decision questions
| ||Reservoir scenario||Fruit scenario|
|Single-value forecast conditions||1 in. of rain||Low temperature of 37 °F|
| ||2 in. of rain||Low temperature of 35 °F|
| ||3 in. of rain||Low temperature of 33 °F|
|Range forecast conditions||2–4 in. of rain||Low temperature of 32–34 °F|
| ||1–5 in. of rain||Low temperature of 31–35 °F|
|Percentage-chance forecast conditions||5% chance of 4 in. or more of rain||5% chance of 32 °F or lower|
| ||10% chance of 4 in. or more of rain||10% chance of 32 °F or lower|
| ||20% chance of 4 in. or more of rain||20% chance of 32 °F or lower|
| ||40% chance of 4 in. or more of rain||40% chance of 32 °F or lower|
Note that each respondent received only one of the two scenarios and one of the two cost conditions. For the scenario and cost condition they were given, however, each respondent received all nine forecast conditions (in random order). Thus, the study employs a combination of between- and within-subject design.
While the two scenarios and associated forecast conditions are similar in many ways, they are not directly parallel. Differences include the scenario content, the phrasing of the protective decision, the positive versus negative perspective of the damage thresholds and uncertainty forecast conditions, and the different numerical values in the thresholds and forecasts. A large body of psychology and related research suggests that such differences can affect people's responses (e.g. Tversky and Kahneman, 1981; Kühberger, 1998; Levin et al., 1998; Windschitl and Weber, 1999; Joslyn et al., 2009b). Given this, the primary goal of testing two scenarios is to examine how results are consistent or different across the two, as a first exploration of the extent to which the findings might apply across contexts.
The binary decision questions were motivated by related experimental work in behavioural economics and psychology that examines individuals' decisions (e.g. Kagel and Roth, 1995; Loewenstein, 1999; Hertwig and Ortmann, 2001; Croson, 2005). The questions have elements of both approaches: for example, the presentation of monetary costs and benefits in the scenarios is more similar to economics experiments, while the context provided in the scenarios is more similar to psychology experiments (Croson, 2005; Ariely and Norton, 2007). The survey implementation employed here also differs from the typical implementation of such experiments in a laboratory setting. From an experimental economics perspective, a major limitation of this study associated with the survey implementation is that subjects did not receive real monetary payoffs related to their decisions. Such monetary incentives are often not used in psychology studies, and previous work suggests that in many situations they do not substantially alter subjects' average behaviour. Nevertheless, monetary incentives can influence some types of findings, and they do tend to reduce variability in subjects' responses (Smith and Walker, 1993; Camerer and Hogarth, 1999; Hertwig and Ortmann, 2001). On the other hand, the survey implementation has the advantage of providing a larger, more diverse respondent population than typical experimental implementations, which usually involve smaller samples of students (Fehr et al., 2003; Naef and Schupp, 2009). With these considerations in mind, these findings are informative as a first study of some of the issues examined here, and they can inform future related survey and laboratory-based work.
Regarding the single-value forecast conditions, the authors hypothesized that some respondents would choose to protect when the forecast had not reached the damage threshold, and that more respondents would protect as the forecast became closer to the threshold. Regarding the range forecast conditions, the authors hypothesized that as an extreme value in the range reached and then exceeded the damage threshold, more respondents would protect. Regarding the percentage-chance forecasts, the authors hypothesized that as the likelihood of exceeding the threshold increased, more respondents would protect, and that the responses would provide evidence of decision making consistent with the cost-loss model. The authors also hypothesized that respondents would be less likely to choose protective action in the higher cost condition.