Reliability of self‐report versus the capacity to consent to treatment instrument to make medical decisions in brain metastasis and other metastatic cancers

Abstract Objective To evaluate the ability of persons with metastatic cancer to self‐assess their medical decision‐making capacity (MDC). To investigate this, we compared an objective measure of MDC with self‐ratings and evaluated predictors of agreement. Methods Data were obtained from a cross‐sectional study of metastatic cancer patients at a large academic medical center. Across all standards of MDC, sensitivity, specificity, and reliability using Gwet's AC1 statistic were calculated using the objective measure as the gold standard. Logistic regression was used to evaluate predictors of agreement between the measures across all MDC standards. Results In those with brain metastases, high sensitivity (greater than 0.7), but low specificity was observed for all standards. Poor reliability was observed across all standards. Higher age resulted in higher odds of disagreement for Standard 3 (appreciation) (OR: 1.07, 95% CI: 1.00, 1.15) and Standard 4 (reasoning) (OR: 1.05, 95% CI: 1.00, 1.10). For Standard 3, chemotherapy use and brain metastases compared to other metastases resulted in higher odds of disagreement (Chemotherapy: OR: 5.62, 95% CI: 1.37, 23.09, Brain Metastases: OR: 5.93, 95% CI: 1.28, 27.55). For Standard 5 (understanding), no predictors were associated with disagreement. Conclusions For less cognitively complex standards (e.g., appreciation), self‐report may be more valid and reliable than more cognitively complex standards (e.g., reasoning or understanding). However, overall, MDC self‐report in the current sample is suboptimal. Thus, the need for detailed assessment of MDC, especially when patients are older or used chemotherapy, is indicated. Other studies should be conducted to assess MDC agreement longitudinally.


INTRODUCTION
Medical decision-making capacity (MDC) is a cognitively mediated functional ability referring to the ability to make informed decisions about medical treatment (Gerstenecker, Niccolai, Marson, & Triebel, 2016). Four core standards of MDC have been outlined in the research literature (Marson, Ingram, Cody, & Harrell, 1995) and include expressing choice, appreciation, reasoning, and understanding. MDC represents an important function at all life stages, but particularly in patients with serious illness, such as metastatic cancer. Take, for instance, the standard of understanding and a person with brain cancer, treatment choices include whole-brain radiation therapy and stereotactic radiation. Understanding allows for recall and conceptual knowledge about each treatment option and its associated advantages and disadvantages. Without this knowledge, informed medical decisions cannot be made. MDC has also been described as a sliding scale in which risks of the decision play a role in determining the level of competency (Drane, 1984). For a lower risk and less complex decision, the standard for competency is lower. However, for a higher risk and more complex decision, the standard for competency is higher. Decisions about cancer treatment will likely fall into the latter category, but metastatic cancer patients will also face many other less risky decisions throughout their care. Therefore, it is important to understand the levels of MDC according to various standards in patients with metastatic cancer.
In adults with primary cancer, approximately 9% will experience symptomatic brain metastases (Eichler et al., 2011), representing the most common type of central nervous system tumors in the United States (Ostrom, Wright, & Barnholtz-Sloan, 2018). Brain metastases, like other forms of cancer, are associated with a range of symptoms , many of which occur before beginning treatment (Janelsins, Kesler, Ahles, & Morrow, 2014). Many studies indicate that metastatic cancer in general accounts for 90% of cancer deaths (Chambers, Groom, & MacDonald, 2002;Jean-Pierre & McDonald, 2016). Aggressive treatment strategies are available and often consist of a combination of surgery, radiation, and pharmacologic interventions (Kotecha, Gondi, Ahluwalia, Brastianos, & Mehta, 2018).
However, although medical management increases life expectancy, it is associated with a number of side effects, including cognitive impairment (Janelsins et al., 2014;Jean-Pierre & McDonald, 2016;Lange et al., 2019). In multiple studies, our group demonstrated that this cognitive impairment leads to corresponding impairment in MDC Triebel et al., 2015).
While MDC has been evaluated in persons with metastatic cancer using subjective, objective, and informant measures, the agreement between these measures has not been evaluated. For instance, persons with cognitive decline in metastatic cancer often do not perform poorly on objective measures (Lange et al., 2019). In turn, some persons with metastatic cancer may be mistakenly identified as possessing fully intact MDC when, in fact, they pose risk for making poor and uninformed treatment decisions. A self-reported or informantreported measure may indicate intact MDC based on everyday behavior and general functioning, but performance when faced with an actual decision may be lacking due to subtle cognitive changes not necessarily observable to the patient or informant.
To examine the gap between self-rated MDC and objective MDC, we administered a self-report measure of MDC to a group of persons with metastatic cancer that has spread to the brain and to a group with metastatic cancer that has not spread to the brain. We then compared these self-report ratings of MDC to a performance-based measure of MDC. We also investigated demographic and clinical correlates of agreement between self-reported and objectively measured MDC in both groups. We hypothesized that self-report would be less reliable in determining MDC compared to the objective measure.

Participants
Newly diagnosed (i.e., first diagnosis and first treatment) persons with metastatic cancer were recruited from the Departments of Radiation  (Shaw et al., 2000). A majority of patients had previously received chemotherapy. Twenty-nine patients were actively receiving chemotherapy at the time of this study.
Overall, the entire battery required about 2 hours to complete. No participants refused to complete the test battery. Trained psychometrists administered the study measures and quality control was conducted by a board-certified neuropsychologist (KT). Medical and treatment data were extracted from medical records.

Capacity to Consent to Treatment Instrument
The Capacity to Consent to Treatment Instrument (CCTI) is a reliable and valid performance-based measure designed to evaluate MDC (Gerstenecker et al., 2016;Marson et al., 1995;Triebel et al., 2015) using five legal standards: Standard 1: expressing a treatment choice orally and in writing. Following initial presentation, the written form is withdrawn and a series of standardized oral questions are asked to assess the four core standards of consent (expressing choice, appreciation, reasoning, and understanding) (Grisso, 1986;Marson et al., 1995).

Current Medical Decision-Making Capacity Rating Self-Report Version
The Current Medical Decision-Making Capacity Rating (CMDC) Self-Report Version is a recently developed self-report measure of MDC.
Participants are presented with an example treatment scenario and then queried using seven questions to determine their perception of their own overall MDC and their MDC within each specific standard. For example, Question 3 asks, "Are you able to appreciate the personal consequences of making a medical treatment choice? That is, do you understand the possible consequences to you personally of one choice versus the other choice?" This question aims to evaluate a participant's assessment of their own ability in Standard 3 (appreciation). Answer choices for this and similar questions evaluating overall MDC and specifically Standards 4 and 5 (reasoning and understanding, respectively) include "Yes-without any help," "Yes-but I need help with this," and "No-I am unable to do this." For this analysis, the latter two responses were considered as "impaired" on self-reported MDC. The measure is included in the Supplemenary material.

Demographics
Self-reported age, gender, race, years of education, and marital status were collected.

Cancer care
Information regarding participants' cancer care was extracted from the medical record by research assistants using standardized data collection forms. Such data included type of primary cancer, radiation and type of radiation, surgical resection, chemotherapy, and hormone treatment.

Medications
Current prescription medications were collected by self-report.

Comorbidities
Potentially relevant comorbidities were also collected via self-report, including diabetes, learning disorders, or head injuries.

Statistical analyses
Bivariate analyses using chi-square tests and t-tests for categorical and continuous variables, respectively, were performed to evaluate differences between those impaired versus those intact on the CCTI within the overall study group and separately for the brain metastasis and other metastasis group. Validity of the self-report of MDC was measured via sensitivity, specificity, positive predictive value, and negative predictive value using the CCTI measure as the gold-standard.
Validity was assessed overall and in each type of metastasis. In this analysis, sensitivity indicated the probability of self-reporting intact given being intact on the CCTI. Specificity indicated the probability of self-reporting impaired given being impaired on the CCTI. Positive predictive value indicated the probability of being intact on the CCTI given self-report of intact, and negative predictive value indicated the probability of being impaired on the CCTI given self-report of impaired.
Additionally, reliability of the self-report measure compared to the CCTI was performed using Gwet's AC1 statistic (Gwet, 2008)

RESULTS
The overall sample consisted of 155 participants with 114 having brain metastasis and 41 having metastasis to other sites. Overall and by metastasis group, the majority of the sample self-reported as  (Table S1). Those intact on Standards 3-5 also had lower total number of neuropsychological tests in the impaired range (Table 1). This pattern followed when conducting bivariate analyses among those with brain metastasis only, but when examining bivariate statistics among those with other metastases, no significant differences were seen for any variables with exception of category fluencies.

Overall study group agreement
Analyses of validity and reliability revealed differences across CCTI standards. Evaluation for Standards 3-5 overall indicated high sensitivity and high negative predictive value (greater than 0.7) for the overall study sample, but low specificity and positive predictive value (Table 2). These results indicate that those identifying as intact on the CCTI were likely to self-report being intact, but those impaired on the CCTI are unlikely to self-report as impaired. Overall for each standard, the self-rated measure correctly identifies participants given they are intact on the CCTI. Sensitivity and specificity results were similar for Standards 3 (appreciation), 4 (reasoning), and 5 (understanding), but for Standard 3, positive predictive value was high and negative predictive value was low (Table 2). When examining reliability, however, the Gwet's AC1 statistic for all groups and standards remained low (Table 2). This indicates that the ability of the self-rated measure to correctly identify participants as impaired or intact is inconsistent compared to the objective measure.

Brain metastasis group agreement
Standards 3-5 overall in the brain metastasis group indicated high sensitivity and negative predictive value (greater than 0.7), but low specificity and positive predictive value (less than 0.7) ( These results again indicate that persons with brain metastasis who test as intact on the CCTI are likely to self-report being intact, but that persons with brain metastasis who test as impaired on the CCTI are unlikely to self-report as impaired. However, concordance between testing results and self-report was lower for the more complex standards of reasoning and understanding. Gwet's AC1 statistic of reliability was low for all Standards, combined and separately, and indicated low reliability/consistency for the self-report measure compared to CCTI (Table 2).

Other metastasis group agreement
The other metastasis group revealed high sensitivity, but low specificity, positive predictive value, and negative predictive value for Standards 3-5 overall ( the other metastasis group exhibited higher reliability and sensitivity, but lower specificity than the brain metastasis group.

Predictors of agreement
When examining predictors of agreement for the overall study group, a 1-year increase in age resulted in 7% increased odds of disagreement for Standard 3 alone (95% CI: 1.00, 1.15) and a 5% increased odds of disagreement for Standard 4 alone (95% CI: 1.00, 1.10). For Standard 3 (appreciation) alone, chemotherapy use resulted in a 5.62-fold increased odds of disagreement, but this estimate had large variability (95% CI: 1.37, 23.09). Also for Standard 3, brain metastasis resulted in a 5.93-fold increased odds of disagreement compared to other metastases, but again this estimate had wide variability (95% CI: 1.28, 27.55).
For Standards 3-5 overall and in Standard 5 (understanding) alone, no predictors were associated with disagreement (Table 3).

DISCUSSION
MDC has been studied across several diseases, but, to our knowledge, no studies have examined the ability of persons with brain metastases to report and recognize their own MDC. To address this gap, the current study evaluated self-report of MDC among persons with metastatic cancer. Investigations into the insight of MDC in persons with metastatic cancer are important because they can help clinicians in determining how much weight to give self-reports of MDC. Overall, our results showed that self-reports were primarily valid only when a participant demonstrated intact MDC on an objective measure. For those with impaired MDC, concordance between self-report and actual MDC performance was low. Taken together, these results indicate suboptimal reliability of our self-report measure. Said another way, as a whole, self-report of MDC in our sample of persons with metastatic cancer was a poor indicator of actual MDC performance.
Although overall reliability of our self-report measure was low, reliability varied according to the cognitive complexity of each MDC standard. For the less cognitively complex Standard of appreciation, reliability of self-report was high. However, for the more cognitively complex Standards of reasoning and understanding, participants were less likely to correctly self-report as intact. Thus, patients may be basing judgment of their own MDC on less complex tasks, such as the ability to remember presented information, while possessing less insight into difficulty with more complex MDC tasks, such as being able to logically understand presented information. Thus, these results imply that simply asking a participant with cancer metastasis, regardless of site, whether they feel confident in their ability to make their own medical decisions may not be sufficient and more formal measures, such as the CCTI, may be warranted. While this requires a psychological or psychiatric referral, it is important to ascertain whether participants fully understand treatment options and their implications both from a legal and an ethical standpoint.
Although both study groups demonstrated difficulty correctly identifying level of performance on the CCTI, an interesting pattern was observed across our study groups. For the other metastasis group, the ability to correctly identify as intact was better than for the brain metastasis group. However, the brain metastasis group was slightly better than the other metastasis group at correctly identifying as impaired. These findings in the other metastasis group are not surprising considering the cognitive impact associated with brain metastasis (Gerstenecker et al., 2014). However, the finding in the brain metastasis group was unexpected. This likely indicates that persons with brain metastasis possess some awareness that their condition is detrimental to cognitive ability and ultimately the ability to make sound medical decisions. Conversely, the more subtle cognitive deficits seen in persons with metastasis that has not spread to the brain (Lange et al., 2019) may lead to a false sense of security when it comes to estimating MDC in the patient group.
Overall, self-report was more reliable in the other metastasis group than in the brain metastasis group. However, specificity remained low even in the other metastasis group, indicating that self-report of MDC may not be valid in metastatic cancer, regardless of spread to the brain.
Moreover, when analyzing the study sample as a whole, most study participants self-identified as intact, and this may have driven some of the discordance seen in the results where many who were impaired on the CCTI self-rated as intact simply due to the high proportion of intact self-rating in the overall sample.
The potential impact of demographic and clinical correlates on the ability to accurately self-report MDC was also analyzed. First, for appreciation and reasoning, increased age was associated with poorer ability to accurately self-report true MDC performance. This could be due to older adults' greater desire for independence, decreased likelihood to admit a problem, or even anosognosia (Fox, Mitchell, & Booth-Jones, 2006;Meyers et al., 2004). Older adults are also likely to have previously undetected age-related cognitive decline and lower education, both of which may influence their ability to self-recognize MDC (Han et al., 2016). Second, prior chemotherapy use demonstrated a higher odds of disagreement between the self-report and CCTI measures for Standard 3. This is not unexpected due to common cognitive side effects of chemotherapy (Du, Xia, & Hardy, 2010;Hurria et al., 2006;Vannorsdall, 2017;Vega, Dumas, & Newhouse, 2017;Wefel, Saleeba, Buzdar, & Meyers, 2010;Wefel & Schagen, 2012). Finally, the predictors must be viewed in the context of our high rate of intact selfratings in the sample. This may indicate that age and past chemotherapy are actually even stronger predictors of MDC than indicated in this sample.

Study strengths and limitations
This study has several strengths. First, this study is the first of its kind to examine inter-rater reliability between a self-report and objective  baum, 2010;Walaszek, 2009). In the current study, we aimed to assess the accuracy of self-report versus objective measures of MDC in less cognitively impaired individuals where assessment is less difficult, but assessment of agreement among individuals with cognitive impairment is important and future studies should examine the impact of cognitive impairment on agreement between self-report and objective measures of MDC. Additionally, sample size limited the assessment of many neuropsychological predictors. We assessed a memory task (HVLT total recall t-score), an executive functioning task (TRAILS B t-score), and a variable indicating the total number of impaired neuropsychological tests for each participant. Future larger studies should examine several neuropsychological tests and domains as predictors. We did not examine agreement longitudinally, which may change over time. Moreover, the CCTI objectively assesses MDC but is not a universally used measure and is not meant to be a substitute for clinical judgment at this time.

CONFLICTS OF INTEREST
The authors report no conflicts of interest.

TRANSPARENT PEER REVIEW
The peer review history for this article is available at https://publons. com/publon/10.1002/brb3.2303