Understanding of an aggregate probability statement by patients who are offered participation in Phase I clinical trials
There is concern that patients with poor numeracy may have difficulty understanding the information necessary to make informed treatment decisions. The authors sought to characterize a special form of numeracy among patients with advanced cancer who were offered participation in Phase I oncology clinical trials.
Surveys were administered to 328 cancer patients who were considering Phase I trials. Their frequency-type numeracy was assessed using a multiple-choice question involving a hypothetical scenario in which a physician stated that an experimental treatment would control cancer in “40% of cases like yours.” In univariate and multivariable analyses, patient characteristics that were associated with better numeracy were identified.
The correct frequency-type interpretation was selected by 72% of respondents. Fourteen percent of respondents incorrectly selected a belief-type answer, “The doctor is 40% confident that the treatment will control my cancer.” In a multivariable model, patients who answered incorrectly tended to have less formal education and less experience with experimental therapies.
Because the misunderstandings some patients demonstrated may influence their treatment decision making adversely, it is critical to identify such patients and to give them special consideration when communicating information about potential risks and benefits of treatment. Cancer 2004. © 2004 American Cancer Society.
Greater involvement of patients in treatment decisions and increasing concern about the quality of the informed consent process have led researchers to examine how well patients understand information about their treatment options.1–11 A particular concern is that some patients do not have the requisite skill, numeracy, to understand and manipulate quantitative information about uncertain outcomes. For example, in a study of highly educated adults, Lipkus et al.12 found low levels of numeracy, with no more than one-third of respondents able to answer all 10 items correctly on the authors' numeracy scale. Others have documented similar levels of numeracy.13 Those studies raise concerns about the ability of patients to understand the information necessary to make informed decisions.
We sought to characterize the numeracy of patients with advanced cancer who were offered participation in Phase I oncology clinical trials. Such patients must be able to understand quantitative information about potential treatments, because they face the daunting task of choosing from among standard therapy, supportive care, and an experimental therapy that offers little chance of benefit. Our previous work with patients considering participation in Phase I trials revealed several opportunities for misunderstanding, including very high patient expectations of benefit from experimental therapies14 and disagreements between patients and their physicians regarding recollections of the information they discussed.14–16 An inability to interpret correctly important, quantitative information regarding treatment options may be another potential source of difficulty in the decision-making process.
Previous research has explored aspects of numeracy that are relevant to patients' ability to manipulate probability information. For example, Schwarz and Sudman17 asked patients various questions that required them to translate percentages into frequencies, and vice versa. Lipkus et al.12 used similar types of questions. Another aspect of numeracy, however, relates to the individual's ability to differentiate between probabilistic statements about relative frequencies (i.e., frequency-type statements) and statements concerning the certainty that a particular event will occur (i.e., belief-type statements).18 Part of understanding frequency-type probability information is knowing which conclusions the information does not allow. This is relatively straightforward with mortality information. For example, if someone is told that the 5-year mortality rate is 3%, then few individuals would interpret the statement to mean that a particular patient will lose 3% of her life by 5 years: By 5 years, the patient will be either dead or alive. With morbidity outcomes, however, there is potential for confusion. For example, if a patient is told that past studies have shown a 5% benefit from experimental therapies, the patient could interpret the statement to mean that the therapy will reduce tumor size for everyone by 5%.
Misinterpretations regarding potential outcomes of various treatment options could have important effects on patients' decision making. For example, interpreting a small chance of benefit to mean that some small benefit (e.g., small tumor reduction) will be enjoyed by everyone who receives the experimental therapy may lead a patient to select an experimental therapy over supportive care. The patient may make a different decision if he or she understands that most individuals will receive no benefit from experimental therapy.
No extant measures of numeracy explicitly assess patients' understanding of this important distinction between frequency-type and belief-type probabilities. Accordingly, we sought preliminary data concerning this distinction by using a simple, single-item measure. We showed patients a clear frequency-type statement and asked them to select the answer that best expressed the statement's meaning. Included among the possible answers were the correct frequency-type interpretation and several belief-type interpretations. Our primary objective was to characterize the frequency-type numeracy of patients who were considering participation in Phase I trials. Our secondary objective was to identify patient characteristics associated with poorer numeracy.
MATERIALS AND METHODS
Patients and Procedures
We used data from a larger study on decision making among patients considering participation in Phase I trials.16 Eligible patients were adults with cancer who had been offered participation in Phase I studies. We excluded patients who already had initiated treatment. Eligibility criteria included 1) advanced malignancy for which there was either no standard effective therapy or for which standard therapy had failed, 2) age ≥ 18 years, 3) life expectancy ≥ 3 months, and 4) an Eastern Cooperative Oncology Group (ECOG) performance status of 0–2 (i.e., the patient was ambulatory at least 50% of the time). We consulted protocol office enrollment lists to confirm the completeness of ascertainment of patients choosing to participate in Phase I studies. Physician consent was obtained prior to patient contact, and written consent to participate in the study was then obtained from patients. Recruitment took place over 18 months, and patient surveys were conducted either in person or by telephone. The Institutional Review Boards of the participating study sites and the data center (Fox Chase Cancer Center, Philadelphia, PA; Duke University Medical Center, Durham, NC; University of Maryland, Baltimore, MD; Georgetown University, Washington, DC; and Northwestern University, Chicago, IL) approved the study design and survey instruments.
A multidisciplinary team, including medical oncologists, nurses, psychologists, clinical economists, and a medical ethicist, developed the questionnaires. Domains were developed based on study objectives, literature review, pilot tests, and relevant past studies.19–29 The survey measures used for the analysis are described below.
We examined age, race/ethnicity, education level, and income to determine whether there were basic demographic or socioeconomic correlates of numeracy. Patients reported their own birth date, gender, race/ethnicity, education level, marital/partnered status, income, and living situation. Due to small cell sizes, we recoded several of the demographic variables. We combined the two lowest education levels (eighth grade or less and some high school), so that education is reported as four levels—some high school or less, high school graduate, some college, or college graduate. We recoded living situation as a dichotomous variable (with 1 indicating living alone and with 0 indicating other) and marital/partnered status as a dichotomous variable (with 1 indicating married/partnered and with 0 indicating other). We recoded monthly income into 4 groups: < $2000, $2000–3999, $4000–5999, and ≥ $6000.
A single item assessed how well patients understood a statement about the relative frequency of benefit from a hypothetical treatment, as follows: “The following question involves a hypothetical situation in which your doctor is describing a new treatment. Imagine that your doctor says this new treatment controls cancer in 40% of cases like yours. How do you interpret what the doctor is saying?” Using a multiple-choice format, patients selected the statement that best described their interpretation of what the physician meant. The correct response was the second of the 7 options listed—“For every 100 patients like me, the treatment will work for 40 patients.” The other 6 options were as follows: “The doctor is 40% confident that the treatment will control my cancer”; “the new treatment will reduce my disease by 40%”; “I am not sure what this information means”; “other (please describe)”; “refused”; and “don't know/unsure.”
Previous therapies and acceptance of Phase I trial enrollment.
Patients with previous involvement in clinical trials may have been exposed to more information about risks and benefits than other patients, so we explored whether previous experience in clinical trials related to numeracy. Patients reported their previous cancer therapies by placing a tick mark next to the item on a list of types of cancer treatments. The list included chemotherapy, radiation therapy, surgery (not biopsy), immunologic therapy (e.g., interleukin-2, vaccines, or antibodies), experimental therapy (e.g., clinical trial), and hormone therapy (e.g., tamoxifen, leuprolide). Patients also indicated whether they had accepted or declined participation in a Phase I study.
Expectations of therapy.
Because some researchers believe that patients who participate in Phase I trials do so because they do not understand information about benefit (see Agrawal and Emanuel1 for review), we investigated whether patients with poorer numeracy had higher expectations of benefit from therapy. To measure expectations of experimental and standard treatments, we asked respondents to place a mark on horizontal bars ranging from 0% to 100% to describe the likelihood of health “benefit” (i.e., the probability that a treatment would “control your cancer”) and “harm” (i.e., the probability that a treatment would cause a “severe adverse reaction”) associated with both standard and experimental therapies.
We handled missing data using multiple imputation by means of a full Bayes multivariate normal imputation model30 that contained all study variables. We implemented this model using the MI procedure in SAS software (SAS Institute, Cary, NC). We computed the fraction of missing information for each of the key study variables.
We summarized characteristics of the unimputed sample using percentages, means, and standard deviations. For all other analyses, we used the multiply imputed data set. We summarized the responses to each multiple-choice option of the numeracy question as percentages. For all subsequent analyses, we recoded the numeracy responses as either correct (coded “1”) or not correct (coded “0”), with the latter category including patients who refused to answer or did not know what the information meant. We calculated descriptive statistics (percentages, means, and standard deviations) separately for patients whose numeracy response was correct and for those whose response was incorrect. We then used univariate logistic regression to estimate the unadjusted odds ratio and associated 95% confidence interval (95% CI) describing the relation between each patient characteristic and numeracy.
Because many patient characteristics were correlated with each other, we conducted multiple logistic regression analysis to estimate the unique and independent relation between patient characteristics and numeracy using the adjusted odds ratio. This approach is helpful in understanding, for example, how differences in numeracy by race/ethnicity may reflect corresponding differences due to education level. We included patient characteristics that had a significant relation to numeracy (95% CI excluded 1.00) in a backward-stepping variable selection process to arrive at a model using the first imputed data set. We then estimated the final model for the multiply imputed data set, from which we obtained adjusted odds ratios and 95% CIs. We determined model fit using the Hosmer–Lemeshow goodness-of-fit test and summarized the model's predictive ability using the c-statistic: Both were calculated for the first imputation sample only.
Of the 593 patients who were asked to participate in the study, 328 patients (55%) participated (184 men and 144 women). The amount of missing information due to missing data ranged from 0.6% to 9.4%; the highest rate of missing data was for the education variable. Table 1 summarizes the characteristics of the sample. The sample consisted largely of middle-aged, well educated, non-Hispanic white patients, most of whom had not received previous experimental therapy but who had decided to participate in a recently offered Phase I clinical trial.
Table 1. Patient Characteristicsa
|Age in yrs (mean ± SD)||57.4 ± 12.2|
|Male gender||184 (56.1)|
| White (non-Hispanic)||279 (85.1)|
| Black (non-Hispanic)|| 39 (11.9)|
| Asian/Pacific Islander|| 4 (1.2)|
| Hispanic|| 3 (0.9)|
| Other|| 3 (0.9)|
| Aleut, Eskimo, or American Indian|| 0 (0.0)|
| ≤ Eighth grade|| 8 (2.5)|
| Some high school|| 18 (5.6)|
| High school graduate|| 89 (27.2)|
| Some college|| 85 (25.9)|
| ≥ College graduate||128 (38.9)|
|Monthly household income|| |
| < $2000|| 74 (22.6)|
| $2000–$3999|| 95 (28.5)|
| $4000–$5999|| 72 (21.8)|
| ≥ $6000|| 89 (27.1)|
|Not living alone||286 (87.2)|
|Types of cancer therapy already received|| |
| Chemotherapy||277 (84.5)|
| Surgery (not biopsy)||154 (47.0)|
| Radiation therapy||144 (43.9)|
| Previous experimental therapy|| 50 (15.2)|
| Hormone therapy|| 21 (6.4)|
| Immunologic therapy|| 21 (6.4)|
|Accepted Phase 1 trial enrollment||260 (79.3)|
|Expectation of benefit: experimental therapy (mean ± SD)||59.4 ± 26.8|
|Expectation of harm: experimental therapy (mean ± SD)||39.8 ± 27.6|
|Expectation of benefit: standard therapy (mean ± SD)||42.1 ± 28.5|
|Expectation of harm: standard therapy (mean ± SD)||55.7 ± 30.0|
Table 2 shows how frequently patients endorsed each response option for the numeracy question. Seventy-two percent of the patients selected the correct answer. Of the remaining responses, the most frequently selected wrong answer was, “The doctor is 40% confident that the treatment will control my cancer” (14%). Approximately 9% of the sample was not sure what the information meant.
Table 2. Frequency of Responses to the Numeracy Question (N = 318 patients)a
|The doctor is 40% confident that the treatment will control my cancer||13.5|
|For every 100 patients like me, the treatment will work for 40 patients||71.7|
|The new treatment will reduce my disease by 40%||2.5|
|I am not sure what this information means.||4.4|
|Refused to answer||0.3|
Whether or not the patients answered correctly was related to several patient characteristics in bivariate analyses (Tables 3, 4). White patients, younger patients, and patients with higher income and education levels were more likely to answer the numeracy question correctly. Participation in previous trials of experimental therapies and previous receipt of chemotherapy and hormone therapy were associated with a greater likelihood of answering correctly. There were no differences in numeracy between patients who did and did not agree to participate in the Phase I trial they were offered at the time of our study. There also was no relation between numeracy and patients' expectations of benefit and harm from standard and experimental treatments (Table 4).
Table 3. Correlation between Numeracy Response and Patient Characteristics—Categoric Variables
|Gender|| || || |
| Male||72.7||1.13 (0.69–1.85)|| |
| Femalea||70.3||1.00|| |
|Race/ethnicity|| || || |
| White||74.6||2.39 (1.27–4.50)b|| |
| Nonwhitea||55.1||1.00|| |
|Education|| || || |
| ≤ High school||37.4||0.08 (0.03–0.22)b||0.07 (0.02–0.20)b|
| High school graduate||59.8||0.20 (0.10–0.41)b||0.19 (0.10–0.39)b|
| Some college||70.0||0.32 (0.16–0.65)b||0.32 (0.16–0.67)b|
| ≥College graduatea||88.0||1.00||1.00|
|Monthly household income|| || || |
| < $2000||56.5||0.26 (0.12–0.58)b|| |
| $2000–$3999||67.5||0.41 (0.18–0.93)b|| |
| $4000–$5999||78.2||0.73 (0.21–2.57)|| |
| ≥ $6000a||83.4||1.00|| |
|Previous experimental therapy|| || || |
| Yes||92.8||6.25 (1.77–22.06)b||7.39 (2.01–27.25)b|
|Previous chemotherapy|| || || |
| Noa||57.3||1.00|| |
| Yes||74.3||2.16 (1.13–4.11)b|| |
|Previous radiation therapy|| || || |
| Noa||68.7||1.00|| |
| Yes||75.4||1.40 (0.85–2.30)|| |
|Previous surgery (not biopsy)|| || || |
| Noa||68.5||1.00|| |
| Yes||75.2||1.39 (0.85–2.29)|| |
|Previous immunologic therapy|| || || |
| Noa||72.3||1.00|| |
| Yes||61.9||0.62 (0.25–1.56)|| |
|Previous hormone therapy|| || || |
| Noa||70.0||1.00|| |
| Yes||95.2||8.55 (1.13–64.72)b|| |
|Accepted Phase I trial enrollment|| || || |
| Noa||69.1||1.00|| |
| Yes||72.3||1.17 (0.65–2.10)|| |
Table 4. Correlation between Numeracy Response and Patient Characteristics—Continuous Variables
|Age in yrs||57.43 ± 12.26||0.80 (0.65–0.98)ab|
|Expectation of benefit: experimental therapy||59.40 ± 26.29||0.99 (0.98–1.00)|
|Expectation of harm: experimental therapy||39.78 ± 27.19||1.00 (0.99–1.01)|
|Expectation of benefit: standard therapy||42.14 ± 27.90||0.99 (0.98–1.00)|
|Expectation of harm: standard therapy||55.68 ± 29.12||1.00 (0.99–1.01)|
The significant variables in the final multivariable model were previous experimental therapy and education level (Table 3). The model achieved acceptable fit (chi-square [degrees of freedom =5] = 2.4; P = 0.79) and predictive ability (c-statistic = 0.75). The adjusted odds ratios for both of these variables were comparable to their unadjusted odds ratios from the univariate analyses.
The ability to understand and manipulate quantitative risk information is crucial for informed decision making in health care. However, recent data from a study by Chao et al.31 suggest that quantitative information about risk and benefit can be confusing, even for relatively sophisticated recipients, such as medical students. Our data address a special type of numeracy among patients who face the decision of participating in Phase I oncology trials. Nearly 75% of these severely ill patients correctly interpreted a statement concerning the aggregate probability of benefit. However, a significant minority of patients interpreted the information about the chance of benefit for a class of patients in ways that were not correct. For example, 14% of patients responded that the statement represented physician confidence rather than a population frequency. Understanding information about the likelihood of benefit is especially important for patients who are considering participation in Phase I trials, because the chance of benefit is low for experimental therapies in the aggregate.1
To optimize informed decision making, it is particularly important to identify patients who are more likely to misinterpret frequency-type probability statements. We found that several patient characteristics were related strongly to numeracy, including race/ethnicity; previous receipt of experimental therapy, chemotherapy, or hormone therapy; education level; income; and age. Given the correlations among the socioeconomic variables, it is likely that the effects of race/ethnicity, age, and income reflect the effect of education. This interpretation is consistent with the fact that none of the variables remained in the multivariable model after adjusting for education. The effect of education was striking: the number of high school graduates who provided the correct answer was 30% lower compared with the number of college graduates. Only 39% of individuals without a high school diploma demonstrated satisfactory understanding. The relation between education and numeracy is not surprising, and researchers should be mindful of this relation during the informed consent process with patients from various educational backgrounds.
Patients with previous experience in trials of experimental therapies had the highest rates of correct response to the numeracy question, even after adjusting for education level and including the use of other therapies in the model. Post hoc analyses indicated that previous experimental therapy was not associated strongly with the use of other therapies; therefore, it was not a simple proxy for cumulative experience with other therapies. One possible explanation for this finding is that patients who agree to participate in clinical trials are more educated and, thus, have higher numeracy. However, this explanation is not consistent with the strong effect of previous experimental therapy after controlling for education level. (In fact, the effect was slightly greater.) Another possible explanation is that patients who had participated in clinical trials were more likely to have tried and failed several treatments, whether standard or experimental. Such patients may have been informed about the chances of benefit in aggregate terms, like those we used in our study. Past failures of experimental and standard therapies may have made those patients more likely to understand that an aggregate chance of benefit means that some individuals will not benefit. Hence, those patients would be more likely to interpret the probability statement in terms of a population rather than in terms of the individual case.
Patients' decisions to participate in Phase I trials and their expectations of benefit and harm from different therapies were not related to numeracy. The latter finding is consistent with the small effect we found in earlier analyses of Phase I trial participants' expectations of benefit14 and may be a result of framing the question about probability of benefit in belief-type terms rather than frequency-type terms. This finding does not eliminate the possibility that some patients who agreed to participate in Phase I trials did so on the basis of a misinterpretation of quantitative information about risks and benefits. Some patients who selected experimental therapy or another treatment option may have made the decision on the basis of such a misinterpretation.
The current findings are relevant to understanding why patients' estimates of their chances of benefit often are so high. We previously described how patients and providers might use probability statements differently, a disparity that can lead to apparent disagreements about chances of benefit.32 For example, a physician might tell a patient that, on average, 5 of every 100 individuals will benefit from an experimental therapy. The patient could then report that the chance that he or she will benefit is 80%. This “discrepancy” is resolved if we understand the physician's statement as a frequency-type statement and the patient's statement as a belief-type statement (i.e., certainty that a particular event will occur). It is not logically inconsistent for a patient to report that 5 of 100 patients will benefit and also to report that he or she is 80% confident of being 1 of those 5 patients.32–34
The current study had some limitations. First, the sample suffered from nonresponse bias, because not all patients agreed to participate in the study. Patients who agreed to participate may have been more interested in research and, thus, may have had a more sophisticated understanding of research-related information. The likely effect of this bias would be an overestimation of the average numeracy of patients considering participation in a Phase I trial. Second, we used only a single item to assess frequency-type numeracy, which results in uncertain reliability of the responses. It should be noted, however, that the responses to the single item did correlate with other study variables, indicating that it has at least a minimally acceptable level of reliability. The use of a single item also may have resulted in a failure to assess all of the nuances associated with an understanding of frequency-type probability. Our group is exploring this issue further through qualitative studies of patient understanding. Finally, although we explored an important aspect of numeracy that has not been studied previously, we did not examine other aspects of numeracy that may be relevant in this patient population, such as the ability to combine probabilities. Future work should examine this issue, perhaps incorporating items from the study by Lipkus et al.12
The current study generated two important findings. First, most patients who were considering participation in Phase I trials were able to interpret the meaning of an aggregate probability statement correctly. In the context of our other work,14–16, 33 this finding contributes to our understanding of whether patients with advanced cancer tend to misunderstand information about the likelihood of treatment benefit. Second, many patients did not interpret an aggregate probability statement correctly. Those patients tended to have less formal education and less experience with experimental therapies. Because the misunderstandings that these patients demonstrated could influence their treatment decision making adversely, it is critical to identify such patients and to give them special consideration when communicating information about potential risks and benefits of treatment. Future research should continue to explore patients' understandings of frequency-type and belief-type probability statements, such that targeted interventions can be developed to assist patients in interpreting treatment outcome information.
The authors thank Damon Seils for editorial assistance and article preparation.