Presented at the Annual Meeting of the Society for Medical Decision Making, Philadelphia, Pennsylvania, October 20, 2008.
The funding agreements ensured the authors' independence in designing the study, interpreting the data, and publishing the report.
The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs.
The authors would like to acknowledge Ellen Peters, Isaac Lipkus, and Mick Couper for helpful discussions and comments; Rosemarie Pitsch for her project management; and Bob Burbach and Aaron Pearlman for creating the risk graphics and for programming, testing, and implementing the survey.
This article is a US Government work and, as such, is in the public domain in the United States of America.
To help oncologists and breast cancer patients make informed decisions about adjuvant therapies, online tools such as Adjuvant! provide tailored estimates of mortality and recurrence risks. However, the graphical format used to display these results (a set of 4 horizontal stacked bars) may be suboptimal. The authors tested whether using simpler formats would improve comprehension of the relevant risk statistics.
A total of 1619 women, aged 40-74 years, completed an Internet-administered survey vignette about adjuvant therapy decisions for a patient with an estrogen receptor-positive tumor. Participants were randomized to view 1 of 4 risk graphics, a base version that mirrored the Adjuvant! format, an alternate graph that showed only 2 options (those that included hormonal therapy), a graph that used a pictograph format, or a graph that included both changes. Outcome measures included comprehension of key statistics, time required to complete the task, and graph-perception ratings.
The simplifying format changes significantly improved comprehension, especially when both changes were implemented together. Compared with participants who viewed the base 4-option bar graph, respondents who, instead, viewed a 2-option pictograph version were more accurate when they reported the incremental risk reduction achievable from adding chemotherapy to hormonal therapy (77% vs 51%; P< .001), answered that question more quickly (median time, 28 seconds vs 42 seconds; P< .001), and liked the graph more (mean, 7.67 vs 6.88; P< .001).
Although most patients will only view risk calculators such as Adjuvant! in consultation with their clinicians, simplifying design graphics could significantly improve patients' comprehension of statistics essential for informed decision making about adjuvant therapies. Cancer 2008. Published 2008 by the American Cancer Society.
One of the most difficult decisions faced by postoperative breast cancer patients concerns whether, and in what form, to take adjuvant therapy to reduce the likelihood of cancer recurrence. This decision involves a tradeoff between the risk reduction achievable by chemotherapy (which is a function of tumor and patient characteristics) and the morbidity associated with these treatments. For patients with estrogen receptor-positive tumors, hormonal therapy is an additional option used either singly or in conjunction with chemotherapy agents. Patients' preferences, specifically the relative value the patient places on reducing the risk of recurrence versus treatment burden, directly influence which choice is optimal.1-3
To help guide decisions about adjuvant therapies, many clinicians use online tools to calculate tailored estimates of mortality risks, recurrence risks, and potential benefits of each therapy option. These estimates are based on complex algorithms that account for (at a minimum) tumor size, grade, and estrogen receptor status, lymph node status, patient age, and patient health status. One of the most commonly used tools, Adjuvant! Version 8 (www.adjuvantonline.com),2-4 presents this information to clinicians (and to patients by means of printable handouts) in a complex graphical format similar to that shown in Figure 1. The graph uses horizontal bars to represent 10-year outcomes for each of 4 possible options as follows: no adjuvant therapy, hormonal therapy only, chemotherapy only, and both chemotherapy and hormonal therapy. The no therapy bar describes how many women out of 100 would be alive (green section), dead because of breast cancer (red section), or dead because of other causes (blue section) in 10 years. The remaining bars repeat this information but also show how the number of women alive would increase with the use of each adjuvant therapy (compared with no therapy) in yellow.
Although this format presents a complete picture of the risks and benefits associated with the adjuvant therapy decision, the risk-communication literature suggests that this graphical format may be suboptimal, inhibiting accurate comprehension of relevant information. Several studies have shown that horizontal bars are more difficult to comprehend than alternate formats like pictographs (sometimes called icon arrays or image matrices).5-10 In addition, the standard Adjuvant! format always displays information about 3 treatment options, each compared with a no therapy option, although in most cases the therapeutic decision is only between 2 options (eg, between hormonal therapy only vs combined therapy when the patient is estrogen receptor-positive or between chemotherapy and no therapy when the patient is estrogen receptor-negative). Such extraneous information increases the cognitive effort required to interpret the graph, which may, therefore, result in reduced understanding.11-13
Risk calculators such as Adjuvant! are designed for use by clinicians, and practicing oncologists are undoubtedly able to correctly interpret the complex graphic with regular exposure. However, many clinicians use Adjuvant! as a tool to facilitate discussion of adjuvant therapy options with their patients, either by presenting and discussing the patient handout or by going online with the patient during a consultation. Because patients lack specific experience with the Adjuvant! tool, their understanding of the risk information may be inhibited by the complexity of the graph.14 If so, patients' misinterpretations of the risk statistics could bias their subsequent adjuvant therapy decisions.
In this study, we tested whether presenting the possible outcomes of different adjuvant therapy options in alternate formats would improve comprehension of relevant risk statistics as compared to presenting the same information in the format currently used in the Adjuvant! tool. We used a randomized experimental design to systematically vary how the risks and benefits of adjuvant therapy options were displayed in a short hypothetical vignette presented to a demographically diverse population of middle-aged and older women participating in an Internet-administered survey. This methodology holds constant the specific risk numbers being displayed. It thus allows direct identification of the effect of different graphical formats without us having to adjust for the variation in prognoses associated with actual cancer patients' diverse tumor characteristics. It also narrows the field of possible graphical formats to a specific recommended format which can be validated in future research using a patient population.
MATERIALS AND METHODS
Overview of Study Design
Each participant was asked to imagine being diagnosed with breast cancer after a routine mammogram. The scenario described surgical removal of the tumor and then presented different options for adjuvant therapy. We randomly varied the format of the graph used to present the mortality risks associated with different adjuvant treatment options and then assessed participants' knowledge of the risk statistics and their preference ratings for the graph type shown. In addition, to measure ease of use, we electronically timed how long participants spent on a key knowledge question. This design received institutional review board exempt-from-approval status, as the design was anonymous survey research.
Study participants were women aged 40-74 years who were drawn from a panel of Internet users administered by Survey Sampling International (SSI) and who voluntarily agreed to receive invitations to fill out questionnaires. E-mail invitations were sent to a stratified random sample of panel members with the goal of approximating the US census on education level, race, and income in the final subject pool. To ensure at least moderate demographic diversity (but not representativeness) and to offset large expected variations in response rates (especially for African Americans and Hispanic Americans), we established target response levels roughly matching the prevalence of these racial and/or ethnic groups in the US population. We also drew 3 distinct age samples within each race (one-third each aged 40-49 years, 50-59 years, and 60-74 years) to offset differential response rates across age groups. The number of E-mail invitations in each demographic subsample was dynamically adjusted until all quotas were achieved, such as requiring at least 180 completed surveys from both the African American and Hispanic American subgroups. Upon their completion of the survey, participants were entered into both an instant contest and a monthly drawing for modest cash prizes administered by SSI.
In our scenario, the respondent was asked to imagine going for a routine mammogram, finding a lump, having a biopsy, and being diagnosed with breast cancer. Respondents were then told that the tumor was removed by surgery (although it was undefined whether the surgery was breast conserving or a mastectomy) and told that the tumor tested as estrogen receptor-positive (but no other tumor characteristics). The scenario then described the physician as making a strong recommendation that the patient take hormonal therapy but leaving up to the patient the question up of whether or not to also take chemotherapy. Respondents then viewed the target graphic along with explanatory text.
To create the graphs, we used mortality risk statistics derived from Adjuvant! for a 59-year-old patient in good health with a 2.5 cm grade 3 estrogen receptor-positive tumor but without lymph node involvement. All study participants received identical risk information. We used a randomized experimental design (subjects were randomly assigned by computer to 1 of 4 experimental conditions) to compare the format used in Adjuvant! (Fig. 1) versus 3 alternative graphs that varied either the format used to display the risk statistics, the number of adjuvant therapy options shown, or both.
Our base graph (Fig. 1) replicated the horizontal stacked bar format used in standard Adjuvant!, with similar layout, proportions, colors, and legend text as printed on the Adjuvant! handout page for patients. Our alternative pictograph format used 10 × 10 matrices of small rectangles to represent possible outcomes. (See tools.cbdsm.org for examples.) The graphic included 4 pictographs, 1 for each treatment option, arranged with the no therapy graph on the left and the 3 adjuvant therapy options to the right. Overall image size was approximately the same as the bar format, and the same color scheme was used to represent outcomes. Consistent with our previous work on the communication of incremental risks,7 however, we modified the legend text to read, “X percent more women of 100 are alive because of therapy.”
Number of options shown
In addition to the 4-option bar graph and pictographs described above, we also created 2 simpler graphs, one in each format, that only displayed 2 bars or pictographs rather than 4. Because the physician in our scenario strongly recommended hormonal therapy, the 2 critical options that respondents needed to consider were hormonal therapy only and combined therapy. In the simpler 2-option graphs, only those 2 options were displayed. In addition, because the no therapy option was omitted, the entire chance of remaining alive was now colored green in the hormonal therapy bar or pictograph, and the yellow incremental benefit area on the combined therapy graph was recalculated to show the marginal increase in survival versus hormonal therapy (instead of vs no therapy). Doing so clarified the pragmatic meaning of the graph by removing extraneous information11 and displayed the incremental benefit in a format that is more easily interpreted using basic graphical perception tasks.15 The 2-option pictograph image, which illustrates both manipulations, is shown in Figure 2.
Our primary outcome measures were 3 questions that assessed respondents' ability to accurately report key statistics relevant to the adjuvant chemotherapy decision as follows: the chance that the respondent would be alive in 10 years with hormonal therapy only, the chance the respondent would be alive with both chemotherapy and hormonal therapy, and how many fewer women out of 100 would die from cancer if they received both chemotherapy and hormonal therapy instead of hormonal therapy only. Because exact numerical information sufficient to calculate these answers was provided in the graph legends, responses were only coded as accurate when they were exactly correct.
We also gathered data on 2 secondary outcome measures. First, as a measure of the cognitive information processing required to interpret the graphs,16 we electronically recorded the number of seconds that the respondent took to answer the risk difference question, which was on a separate page from all other questions. Second, we asked respondents to provide 3 perception ratings about the graph they saw. Answering each question on a 10-point scale, respondents rated how well the graph described the benefits of different additional treatments, whether the respondent would prefer to see risk information in this type of graph, and how clearly the graphs represented the increase in the chance of being alive.
Individuals vary in terms of their numeracy, ie, their facility and comfort with quantitative health information such as risk statistics. To assess this important covariate, study participants completed the Subjective Numeracy Scale (SNS),17, 18 a validated measure of quantitative ability and of preferences for receiving information in numerical form. The SNS comprises 8 questions, 4 assessing perceived numerical ability (eg, “How good are you at calculating a 15% tip?”) and 4 assessing preferences for quantitative information (eg, “How often do you find numerical information to be useful?”). SNS scores range from 1 (least numerate) to 6 (most numerate). The SNS has previously been shown to correlate with the ability to recall and comprehend risk communications in both textual and graphical formats.18
In addition, participants completed demographic measures including level of education. For analysis purposes, we modeled education as a 3-level variable as follows: high school (HS) or less, some post-HS education but no Bachelor's degree, and Bachelor's degree or more.
On the basis of prior research that used pictographs,5-10 we expected that this format would facilitate study participants' efforts to comprehend the risk information provided. Thus, we predicted that women who were shown risk information in pictograph form would be both more accurate on the comprehension questions and quicker to complete the accuracy tasks than women shown the horizontal bar format. Because of these advantages, we also hypothesized that respondents would rate pictographs as a more preferred format than horizontal bar graphs.
Our simpler, 2-option graphs eliminated nonessential information and reframed the incremental benefit to make comparing hormonal therapy only with combined therapy easier. Because these changes facilitate direct comprehension of the risk tradeoff between hormonal therapy and combined therapy,19-21 we hypothesized that respondents who received 2-option graphs would also have increased comprehension accuracy, faster task completion times, and higher graph preference ratings than respondents who viewed 4-outcome graphs.
We used chi-square tests of proportions to test whether the format of a graph affected comprehension of risk statistics, Student t tests to compare graph preference ratings, and Wilcoxon rank-sum tests (to compensate for highly skewed distributions) to compare the distributions of time spent answering the knowledge question on the marginal benefit of treatment. We also used a logistic regression analysis to assess whether participants' comprehension of different graphs was mediated by numeracy. All analyses were performed by using STATA 10,22 and all tests of significance were 2-sided and used α = .05.
A total of 2251 individuals reached the survey website and viewed the first content page. Of these, 603 (27%) failed to complete the survey. In addition, 5 were excluded for completing the survey too quickly to have paid attention, 16 were male and hence excluded, and 8 were excluded for reporting ages outside of the requested sample range. (See Fig. 3 for details of participant flow through the survey instrument.) Completion rates did not differ significantly across the 4 arms of our randomized controlled trial design. Our analyses focus on the remaining 1619 participants.
Sample demographic characteristics are described in Table 1. We observed a wide range of educational achievement, with 27% of participants having completed a Bachelor's or higher college degree but also 25% with only a high school education or less. While 23% of respondents reported having had a prior breast biopsy, 4% had a prior diagnosis of breast cancer, and 19% reported having a first-degree relative with a prior diagnosis of breast cancer, a sensitivity analysis showed that exclusion of these groups did not qualitatively change the results reported below. As expected given our experimental design, there were no significant variations in sample demographics across experimental conditions.
Table 1. Sample Characteristics
SD indicates standard deviation from the mean.
Subjective Numeracy Score, 1-6
Hispanic ethnicity, any race
≤High school diploma
Prior breast cancer experience
Prior breast biopsy
Prior breast cancer diagnosis
First-degree relative with breast cancer
Comprehension of Risk Statistics
Because our scenario described a patient with an estrogen receptor-positive tumor, our first 2 comprehension questions assessed participants' ability to report the total chance of being alive in 10 years if the patient took hormonal therapy only or if she took both chemotherapy and hormonal therapy. The results are shown in Table 2. The accuracy rates among participants viewing the base 4-option bar graph were strikingly low; approximately 17% of respondents answered each question correctly. However, significantly improved accuracy was observed with each of our alternative graphs, especially the 2-option pictograph.
Table 2. Proportion of Respondents Correctly Reporting Total Survival Rates by Graph Type
Perhaps the most critical information related to the adjuvant therapy decision presented in our scenario is the difference between these 2 numbers, ie, the incremental risk reduction achieved by adding adjuvant chemotherapy to hormonal therapy. The percentage of respondents correctly noting that 2 fewer women out of 100 would die if they took chemotherapy in addition to hormonal therapy is shown in Figure 4. Although respondents who viewed 4-option graphs (whether bar or pictograph) were only able to correctly answer this question about half of the time, accuracy was significantly improved among participants who were shown the 2-option bar graph [χ2(1) = 14.95; P< .001] and again especially among those who viewed the 2-option pictograph [χ2(1) = 57.23; P< .001].
A logistic regression analysis (Table 3) showed that these format effects remain highly significant after we controlled for both a strong and statistically significant effect of individual numeracy as well as a weaker independent effect of education. All race, ethnicity, and breast cancer experience variables were nonsignificant predictors of comprehension. An expanded model (not shown) showed no significant interactions between numeracy and any of the graph formats. As a result, respondents who saw 2-option pictographs had higher comprehension rates than those who viewed 4-option bar graphs, regardless of whether they scored above median on the Subjective Numeracy Scale lpar;2-option pictograph, 85.3% vs 4-option bar, 62.4%) or below median (2-option pictograph, 69.0% vs 4-option bar, 43.1%).
Table 3. Logistic Regression Analysis of Respondents' Comprehension of the Risk Reduction due to Addition of Chemotherapy to Hormonal Therapy
Comprehension of Benefit of Adding Chemotherapy to Hormonal Therapy
Base graph is a 4-option horizontal bar graph. Number of respondents was 1568.
The median time spent completing the risk-difference question, which was asked on a separate web page from the rest of the survey, is shown in Figure 5. Presenting all 4 treatment options in a pictograph instead of the base horizontal bar format had no effect on time spent. However, Wilcoxon rank-sum tests showed that respondents who viewed simplified images that only presented information about 2 options were able to complete the question in significantly less time (bar, z = 4.69; pictograph, z = 7.89; both P< .001), with the shortest median time (28 seconds) observed in the group who received the 2-option pictographs. In addition, among respondents who viewed the 2-option graphs (but not among respondents who viewed 4-option graphs), knowledge accuracy was significantly higher among participants who completed the task in 30 seconds or less (bar, 78.2%; pictograph, 84.0%) compared with participants who took more than 30 seconds to complete the task [bar, 54.0%, χ2(1) = 25.82, P< .001; pictograph, 68.6%, χ2(1) = 12.53, P< .001].
Ratings of Different Formats
Participants ratings on the 3 graph-perception questions were highly correlated, so we combined all 3 questions into a single scale with very high reliability (α = .91). The 4-option and the 2-option pictograph graphics received the highest scores (mean = 7.68 & 7.67, respectively), significantly higher than those for the base 4-option bar graph (mean = 6.88; Student t = 4.62 vs 4-option pictograph; Student t = 4.43 vs 2-option pictograph; both P< .001). The 2-option bar graph was also significantly preferred to the 4-option base graphic, although the effect was not as large (mean = 7.33 vs 6.88; Student t = 2.49; P = .01).
Whereas decision support tools such as Adjuvant! use graphical displays to communicate the mortality risks that patients face with different adjuvant therapy options, our research shows that women had difficulty interpreting the 4-option horizontal bar format currently used by Adjuvant!. Two simple changes, displaying only risk information related to treatment options that included hormonal therapy (because the scenario described an estrogen receptor-positive tumor) and using pictographs instead of horizontal bars, resulted in significant improvements in both comprehension accuracy and speed of use in our demographically diverse sample. Furthermore, respondents showed strong preferences for pictograph formats over the currently used horizontal bar format. It is important to note, however, that sizeable knowledge deficits were still observed even when risk information was presented in the best format tested in this study, the 2-option pictograph. Further research is clearly needed to explore even more simplified formats to determine whether we can further improve patient understanding of the risk tradeoffs associated with adjuvant therapy decisions.
We draw particular attention to the finding that the participants who viewed the 2-option pictographs not only took the least amount of time to complete knowledge tasks (Fig. 5) but also had the lowest error rates (Fig. 4). Together, these data demonstrate that reading and interpreting the 2-option pictographs required less cognitive effort than the 4-option bar graphs. The task of making complex treatment decisions is both cognitively demanding and emotionally stressful, and evidence suggests that decision-making performance is often degraded under such conditions.23, 24 Moreover, studies have shown that cognitive effort induces negative emotions in many people and that these emotions can cause them to withdraw from making decisions.25 Thus, even if patients could figure out more complex graphics given time and support from their clinicians, their ability to use this information to make their decisions would be impeded by the cognitive effort required to obtain it. The use of simpler graphical formats may help to offset this unwanted effect.26, 27
Although individual numeracy levels were strongly predictive of risk knowledge, the design of the risk graphic affected both high numerate and low numerate individuals similarly. Such findings reinforce our belief that optimal design of risk graphics is essential for all users, not just for those less educated or less numerate.
Our research has several limitations. First, although our Internet sample contained substantial demographic diversity, we did experience some significant dropout during the survey. Those individuals who failed to complete the survey (and hence did not provide comparable demographic information) may have had different characteristics from those who completed it. Our participants may also be nonrepresentative in unidentified ways (for example, because they enjoy taking surveys). However, we ensured internal validity by using an experimental design. Furthermore, our previous research using this panel has shown that Internet survey responses from this panel closely match those of representative samples.28 Second, most participants (75%) reported having had at least some education beyond high school, a trait which may limit our ability to generalize these findings to a less educated population. Third, our scenario was entirely hypothetical, and actual cancer patients may be more motivated to correctly interpret risk graphics presented to them by clinicians. Patients also have the opportunity to discuss such graphs in face-to-face consultations with their oncologist, which undoubtedly leads to better comprehension than we observed. Nevertheless, our experimental results suggest that the use of nonoptimal risk-communication graphics can significantly inhibit comprehension of key statistics, whereas simpler graphics may enable clinicians to spend less time explaining risk information to patients and more time discussing its implications for each patient's adjuvant therapy decision.
The results presented here support the concept that simpler information displays can make it easier for decision makers to implement optimal decision strategies.23 Specifically, focusing patients' attention on those treatment options currently under consideration while removing information related to options that have been already eliminated from consideration (for medically appropriate reasons) may be particularly beneficial.25, 29, 30 In the context of adjuvant therapy decisions, such an approach would suggest that clinicians should discuss the decision in 2 stages as follows: a first stage in which hormonal therapy is considered and a second stage in which the incremental benefit of chemotherapy is evaluated. The 2-option pictograph tested here would be highly appropriate for the second stage of this discussion, and a similar graphic showing outcomes for no therapy versus hormonal therapy could be used to improve patient comprehension during the first stage of discussion.
Adjuvant! and other online risk calculators enable oncologists and patients to receive individually tailored estimates of mortality and recurrence risks, information that is essential to making informed decisions about adjuvant therapy options. Yet, the full potential of these modeling applications cannot be realized if users misinterpret the statistics provided.14 Our results show that certain graphical formats can preclude patient comprehension. Clinicians may face similar difficulties when considering statistics presented in these formats for clinical decision making. Developers of risk-communication and decision-support tools should incorporate evidence-based, simplifying design elements, such as removal of information not required for the current decision and the use of pictograph formats, into both existing and future tools.