Fax: (901) 495-5845
Reliability and validity of the Childhood Cancer Survivor Study Neurocognitive Questionnaire
Article first published online: 12 SEP 2008
Copyright © 2008 American Cancer Society
Volume 113, Issue 8, pages 2188–2197, 15 October 2008
How to Cite
Krull, K. R., Gioia, G., Ness, K. K., Ellenberg, L., Recklitis, C., Leisenring, W., Huang, S., Stovall, M., Robison, L. L. and Zeltzer, L. (2008), Reliability and validity of the Childhood Cancer Survivor Study Neurocognitive Questionnaire. Cancer, 113: 2188–2197. doi: 10.1002/cncr.23809
- Issue published online: 3 OCT 2008
- Article first published online: 12 SEP 2008
- Manuscript Accepted: 30 MAY 2008
- Manuscript Revised: 27 MAY 2008
- Manuscript Received: 14 MAR 2008
- National Cancer Institute. Grant Number: U24-CA55727
- American Lebanese Syrian Associated Charities
- Childhood Cancer Survivor Study;
- late effects
Up to 40% of childhood cancer survivors may experience neurocognitive impairment in 1 or more specific domains. As such, regular monitoring has been recommended for patients exposed to cranial irradiation and/or antimetabolite chemotherapy. This study reports the results of a questionnaire developed to identify those survivors who may be experiencing neurocognitive problems.
Participants for this study were 7121 members of the Childhood Cancer Survivor Study cohort (6739 survivors and 382 siblings). These participants completed a new neurocognitive questionnaire designed to assess functions commonly affected by cancer therapy, as well as a standard measure of emotional functioning. A measure of cognitive and emotional functioning was also completed on a subset of the patients roughly 7 years before the current questionnaire. Responses to the questionnaires among subgroups of survivors were then analyzed to examine the reliability and validity of the new neurocognitive questionnaire.
Four reliable factors were identified that assessed task efficiency, emotional regulation, organization, and memory skills. These neurocognitive factors accurately discriminated survivors who were at “high risk” for neurocognitive dysfunction, because of neurologic abnormalities or a history of intensive focal cranial irradiation, from healthy “low-risk” survivors and siblings.
The questionnaire demonstrated excellent reliability, as well as construct and discriminative validity. It appears to be a practical and efficient tool for monitoring neurocognitive outcomes in adult survivors of pediatric cancer. Cancer 2008. © 2008 American Cancer Society.
Today in the US, approximately 80% of the nearly 12,000 children and adolescents diagnosed with cancer each year can expect to survive at least 5 years.1 With increased survival, attention has turned to quality of life and functional outcome after treatment, including neurocognitive performance. Studies suggest up to 40% of childhood cancer survivors may experience neurocognitive impairment in 1 or more specific domains (eg, processing speed, attention, memory).2‒5 Impairment in these domains impede the learning of new information, along with maintenance of previously learned information, and ultimately lead to declines in global intellect. This in turn results in poor academic and vocational success, low self-esteem, and behavioral or emotional disorders.
Regular monitoring of neurocognitive outcomes has been recommended for patients exposed to cranial irradiation and/or antimetabolite chemotherapy during treatment of pediatric cancer.6 Ideally, such monitoring would involve routine comprehensive neurocognitive evaluations. However, this approach may not be practical for a variety of reasons, including excessive cost, required time, and limited professional resources. This study reports the results of a questionnaire developed to identifying those survivors who may be experiencing neurocognitive problems.
Questionnaires are an established method for screening a variety of functions. Such scales have been used for many years in the assessment of emotional and behavioral problems in children (eg, the Child Behavior Checklist, Behavior Assessment System for Children), as well as adults (eg, Minn Multiphasic Personality Inventory, Brief Symptom Inventory).7‒10 The assessment of children typically involves completion of ratings by parents/guardians, whereas it is customary to collect self-reports for adults. More recently, the use of rating scales has expanded to aid in the assessment of health-related behaviors or outcomes, such as sleep habits,11 quality of life,12 and neurocognitive function.13, 14 A recent extension of neurocognitive ratings has been the focus on specific executive functions.15, 16
Currently, there are 2 scales designed to assess general neurocognitive functioning in adults. The Cognitive Behavior Rating Scales (CBRS) was standardized in adults aged 30 to 89 years, and generates measures of language deficits, agitation, need for routine, depression, higher cognitive deficits, memory, dementia, apraxia, and disorientation.14 The CBRS focuses on significant impairment and is generally used in the assessment of individuals with dementia. The Neurobehavioral Functioning Inventory (NFI) was standardized on adults aged 17 to 80 years, and generates indices of depression, somatic complaints, memory/attention problems, communication difficulties, aggression, and motor deficits.13 The NFI is typically used in rehabilitation settings for treatment planning and tracking change over time.
There are 2 widely used scales designed to evaluate executive functions in adults. Executive functions are generally defined as abilities required for the planning, implementing, monitoring, and adaptation of behavior for success in one's environment.17, 18 The Behavior Rating Inventory of Executive Function (BRIEF) is a widely used questionnaire designed to assess real-world aspects of executive dysfunction in children.19 The recently released adult version of the BRIEF (BRIEF-A) includes 2 scales: Behavioral Regulation Index (BRI) and Metacognition Index (MI). BRI is comprised of 4 subscales (Inhibit, Shift, Emotional Control, and Self-Monitor), and the MI is comprised of 5 subscales (Initiate, Working Memory, Plan/Organize, Task Monitor, and Organization of Materials).16 The Frontal Systems Behavior Scale (FrSBe) was originally developed for use in adults and includes 3 major scales: Apathy, Disinhibition, and Executive Dysfunction.15
The utility of these scales in the assessment of functioning in long-term survivors of childhood cancer is uncertain. When neurocognitive deficits do occur in long-term survivors, they typically involve problems with attention, processing speed, and/or memory.4, 5, 20 The CBRS and NFI have demonstrated sensitivity to general dysfunction and emotional adjustment in dementia and after acute brain injury.14, 21 However, neither of these scales differentiate between specific neurocognitive skills. The BRIEF-A and FrSBe focus on assessment of executive dysfunction, which may indeed be impacted in long-term survivors. However, neither measure includes assessment of attention, processing speed, or memory. Furthermore, the sensitivity and specificity of these scales have not been evaluated in adults who are long-term survivors of childhood cancer.
The European Organization for Research and Treatment of Cancer has developed a quality of life instrument that includes a measure of cognitive function.22 Although this scale does appear sensitive to the effects of a variety of cancers and therapies,23 the cognitive scale is 1-dimensional, and thus does not separate skills into neurocognitive subdomains that may demonstrate treatment specificity.
The purpose of the current study was to report the utility of a new rating scale developed for the ongoing Childhood Cancer Survivor Study (CCSS).24 This Neurocognitive Questionnaire (CCSS-NCQ) was constructed to assess skills that are reported to be sensitive to the effects of radiation and antimetabolite chemotherapy in long-term cancer survivors, as well as skills included in currently established measures of executive functions.
MATERIALS AND METHODS
Participants for these analyses were members of the CCSS cohort. The CCSS is a retrospective cohort study designed to evaluate the impact of childhood cancer and its treatment on long-term function and health. Details of the criteria for cohort inclusion have been reported previously.24 Briefly, eligible participants were treated for 1 of 8 childhood cancer diagnoses at 26 institutions between 1970 and 1986 when younger than 21 years of age. Cohort entry was limited to those individuals who survived for at least 5 years after their original diagnosis. The human subjects committee at each of the collaborating institutions approved the study protocol before participant enrollment. Participants provided informed consent for the questionnaires and separate consent for medical record abstraction. Medical record abstraction was completed for all consenting participants. Study participants completed a 24-page baseline questionnaire between 1995 and 1996. This questionnaire surveyed sociodemographic information, medical history, and functional limitations. A second follow-up questionnaire, which included the CCSS-NCQ, was completed between 2002 and 2004 (the full survey questionnaire is available at http://www.stjude.org/SJFile/ccss_fu2.pdf; the CCSS-NCQ). Participants in the current analyses completed both the baseline questionnaire and the second follow-up survey, including the CCSS-NCQ. A total of 7121 participants from the original cohort completed the CCSS-NCQ. Diagnoses and demographics are presented in Table 1.
|Characteristic||Survivors (n=6739)||Siblings (n=382)||P|
|<High school graduate||251||3.8||9||2.4|
|High school graduate||3320||49.7||181||47.6|
|Soft tissue sarcoma||613||9.1||NA|
|Age at interview||32.1||7.6||34.1||8.4|
|Age at diagnosis||8.5||5.9||NA|
The CCSS-NCQ was developed by selecting a subset of items represented in an early investigational version of the BRIEF-A, to which additional items were added (the CCSS-NCQ was administered before the publication of the BRIEF-A). In conjunction with the primary author of the BRIEF,19 versions of items were selected to represent the following cognitive constructs: Organization of Materials, Working Memory, Plan/Organize, Initiate, Inhibit, and Emotional Control. To this item list, additional items were added that focused on processing speed, memory, and academic functioning. This resulted in the 25-item CCSS-NCQ. The items were presented with instructions asking participants to report the degree to which they experienced any of the 25 problems over the past 6 months. Possible responses were presented using a Likert scale ranging from 1 (“Never a Problem”) to 3 (“Often a Problem”). Of the participants who completed the CCSS-NCQ, 6739 were survivors, and 382 were siblings.
The Behavior Problem Index (BPI)25 was originally developed for the National Health Survey by taking a subset of questions from the Child Behavior Checklist.26 These questions were completed by parents of adolescents 12-17 years of age during the baseline questionnaire (ie, 1995-1996). For each of the 32 items, parents were asked to describe their child's behavioral and emotional functioning. Possible responses were presented using a Likert scale ranging from 1 (“Not True”) to 3 (“Often True”). The BPI has recently been validated in a CCSS sample.27 In that report, the following 5 factors were identified: Depression/Anxiety, Headstrong Behavior, Attention Deficit, Peer Conflict, and Antisocial behavior. Of those participants who completed the CCSS-NCQ, 1671 parents had completed the BPI during the baseline questionnaire.
The Brief Symptom Inventory-18 (BSI) is an 18-item screening questionnaire designed to assess symptoms of depression, anxiety, and somatic complaints.9 Participants were asked to describe the extent to which they have been distressed or bothered by each symptom in the previous 7 days. Possible responses were presented using a Likert scale ranging from 1 (“Not at All”) to 5 (“Extremely”). This measure was collected in the CCSS follow-up survey at the same time the CCSS-NCQ was obtained. As with the latter scale, 6739 survivors and 382 siblings completed the BSI. A recent report using the CCSS sample has supported the initial factor structure of the BSI in adult survivors of childhood cancer: Depression, Anxiety, and Somatic Complaints.28
To recap, the BPI was administered once during the baseline survey conducted between 1995-1996, whereas the BSI and CCSS-NCQ were each administered once during a follow-up survey conducted between 2002 and 2004.
An exploratory factor analysis was conducted using all reports collected from siblings who completed the 25-item CCSS-NCQ. The maximum likelihood method was used to extract the factors, which were then subjected to a promax (oblique) rotation. Internal consistency of the factors was assessed using Cronbach Alpha.29 A confirmatory factor analysis was then conducted using all reports from cancer survivors. The weighted least squares method was used, as the multivariate normal assumption was in doubt.
Construct validity of the factors was explored by comparing performance on the CCSS-NCQ factors to the BPI items collected on the CCSS baseline questionnaire. Although the BPI was collected roughly 7 years before the CCSS-NCQ, the BPI was administered during adolescence, a period of development when cognitive patterns become engrained. When attention problems persist into adolescence, they typically continue into adulthood and expand to impact executive functioning.30, 31 Thus, parent report of significant problems during adolescent should be reflected in self-reports, even 7 years later. Concurrent validity was examined by comparing the established factor scores to the factors from the BSI, which was also collected during the second follow-up questionnaire. Discriminative validity was examined by comparing the CCSS-NCQ factors between cancer survivors at “high risk” to those at “low risk” for neurocognitive dysfunction. For this purpose 2 “high-risk” groups were defined: 1 group (n = 172) included all survivors with epilepsy and/or cerebrovascular abnormalities (ie, Neurologic group); the other group (n = 247) included survivors treated with high-dose (≥35 Gray [Gy]) cranial radiation therapy (CRT) to the frontal area of their brains (ie, Frontal CRT group). A cutoff of ≥35-Gy CRT was established based on the attempt to identify a “high-risk” group for validation of the instrument. In a recent review of CRT effects on late neurocognitive sequelae, >35 Gy was a level consistently associated with significant impairment.32 Although lower doses of CRT and chemotherapy treatment (eg, 18 Gy CRT + intrathecal [IT] and/or high dose intravenous [IV] methotrexate) present elevated risk, the risk is not consistently perceived to be “high.” Furthermore, at least 2 recent reports question the impact of lower doses of CRT and chemotherapy on neurocognitive functions.33, 34 Neurocognitive dysfunction has also been consistently reported in samples of adults with epilepsy,35‒39 stroke,40‒42 and cerebrovascular abnormalities.43‒45 Increased incidence of such neurologic disorders has been documented in survivors of childhood cancer treated with cranial radiation.46, 47 These “high-risk” groups were compared with a “low-risk” group of healthy survivors who had no history of central nervous system (CNS) disease, no history of CNS treatment, and no history of major organ complication or procedures (n = 688). These groups were also compared with the sibling group (n = 382). Multivariate analysis was done to assess group differences. As a method to evaluate clinical significance of the ratings, performance on each scale was reclassified based on the frequency of an item being rated as “Often a Problem” (ie, the highest possible rating available). The frequency of these ratings were then calculated for the sibling group, and logistic regression was used to evaluate odds ratios (ORs) for the likelihood of higher scores in each scale for each of the “high-risk” and “low-risk” survivor groups compared with the siblings. Robust standard errors were calculated based on generalized estimating equation methods, to account for intrafamily correlation between survivors and siblings, and were used to obtain 95% confidence intervals. All regression models were adjusted for age and sex. Two-sided P values <.05 were considered significant.
During the exploratory factor analysis (EFA) in 382 siblings, 4 factors met criteria for retention based on eigenvalues, assessment of the scree plot, and face validity of grouped items. In interpreting the rotated factor pattern, an item was said to load on a given factor if the factor loading was 0.40 or greater. The factor loadings from the exploratory analysis and the internal consistency estimates of the rotated factors are presented in Table 2. Four factors representing 19 of the 25 variables were obtained: Task Efficiency, Emotional Regulation, Organization, and Memory. The Task Efficiency factor accounted for the major proportion of variance and was comprised of items related to initiating and completing tasks in a rapid manner. Emotional Regulation significantly contributed to the overall variance and was comprised of items related to emotional reactivity and frustration tolerance. The Organization factor was comprised of items related to organization of the individual's environment. The fourth factor, Memory, was comprised of items related to both working memory and long-term memory (ie, recall of information learned previously).
|Question||Task Efficiency||Emotional Tolerance||Organization||Memory|
|Gets upset easily*||−0.07||0.76||0.00||−0.13|
|Takes longer to complete work*||0.64||0.05||0.02||−0.06|
|Does not think of consequences†||0.09||0.19||0.14||0.03|
|Forgets instructions easily*||0.20||−0.06||0.06||0.59|
|Problems completing my work*||0.65||0.01||0.21||−0.14|
|Difficulty recalling things learned before*||0.01||0.04||0.09||0.61|
|Gets frustrated easily*||−0.06||0.74||−0.02||0.15|
|Mood changes frequently*||0.09||0.69||−0.04||−0.02|
|Different ways to solve a problem†||0.24||0.33||−0.06||0.13|
|Forgot what doing in middle of things*||−0.02||0.13||0.13||0.45|
|Problems with self-motivation*||0.41||0.03||0.21||0.10|
|Trouble finding things in bedroom*||−0.13||0.09||0.65||0.07|
|Trouble with multitasking*||0.47||0.13||0.07||0.05|
|Blurts things out†||0.02||0.25||0.11||0.07|
|Desk/workspace a mess*||−0.01||−0.07||0.72||0.02|
|Trouble remembering things||0.06||0.03||−0.03||0.77|
|Trouble prioritizing activities*||0.42||0.06||0.26||0.05|
|Slower than others*||0.73||−0.12||−0.08||0.05|
|Does not work well under pressure*||0.54||0.10||−0.10||0.09|
|Trouble solving math problems†||0.32||0.01||−0.18||0.29|
Confirmatory analysis (CFA) was conducted in the survivor population (n = 6739) using the 4-factor solution from the EFA. The goodness of fit index (GFI) was 0.91, and the root mean square error of approximation (RMSEA) was 0.05. RMSEA is an index of fit that is less influenced by sample size, with recommended levels being 0.05 or below.48 These indices suggest a reasonable fit of the 4-factor structure to the cancer survivor sample. A CFA was also conducted using only those survivors determined to be at “high risk” for neurocognitive dysfunction (ie, the Neurologic group and the Frontal CRT group). The fit index for this at-risk group was acceptable (GFI = 0.93). As part of the consideration of other factor solutions in explaining the pattern of data from the survivors, additional CFAs were conducted. CFA was used with a single factor to determine whether the combined response to all 25 items best explained the variability in survivor responding. This resulted in a significantly reduced fit index (GFI = 0.83). A CFA was also conducted using a 2-factor solution based on the factors from which the represented items on the BRIEF-A were selected (ie, Behavioral Regulation and Metacognition). Again, this resulted in a significantly reduced fit index (GFI = 0.88). Thus, the 4-factor solution produced the best fit during confirmation.
Construct validity of the 4 CCSS-NCQ factors was demonstrated through correlation with the BPI factors. Of the 4 factors derived, Task Efficiency was expected to correlate most highly with the BPI factor Attention Deficit. This was, in fact, the resultant outcome. Table 3 presents correlations between the factors on the 2 rating scales, along with kappa coefficients (κ). Kappa is an index of agreement between dichotomous variables, which attempts to account for base rates in the population, with “fair agreement” being a coefficient ≥0.20.49, 50 The relevance of these correlations and agreement coefficients is increased through recognition that the questionnaires were completed by 2 different raters (ie, cohort completed the CCSS-NCQ, whereas parents completed the BPI) over an interval of roughly 7 years (ie, the BPI was completed between 1995 and 1996, and the CCSS-NCQ was completed between 2002 and 2004). Of note, the Depression/Anxiety factor from the BPI was correlated more strongly with the CCSS-NCQ Emotional Regulation than any of the other NCQ factors. Overall, the pattern of correlations with the BPI provides initial support for the 4-factor constructs associated with the CCSS-NCQ.
|Task Efficiency, r (κ)||Emotional Regulation, r (κ)||Organization, r (κ)||Memory r (κ)|
|Behavior Problem Index|
|Depression/Anxiety||0.25 (0.13)||0.27 (0.19)||0.12 (0.08)||0.20 (0.10)|
|Headstrong Behavior||0.24 (0.10)||0.30 (0.16)||0.15 (0.07)||0.21 (0.10)|
|Social Deviance||0.22 (0.11)||0.27 (0.17)||0.18 (0.10)||0.20 (0.14)|
|Attention Deficit||0.40 (0.21)||0.32 (0.18)||0.25 (0.14)||0.32 (0.19)|
|Peer Conflict||0.32 (0.16)||0.25 (0.16)||0.17 (0.07)||0.26 (0.14)|
|Brief Symptom Inventory|
|Depression||0.44 (0.27)||0.49 (0.34)||0.23 (0.17)||0.35 (0.23)|
|Somatization||0.35 (0.20)||0.38 (0.24)||0.20 (0.13)||0.34 (0.22)|
|Anxiety||0.38 (0.22)||0.49 (0.32)||0.23 (0.12)||0.34 (0.19)|
Concurrent validity was assessed through comparison of responses to the CCSS-NCQ and the BSI, both of which were completed during the same assessment. Here, it was expected that the CCSS-NCQ Emotional Regulation factor would correlate more highly with the BSI factors than would the other CCSS-NCQ factors. Again, as evident in Table 3, this was the resultant outcome. Significant correlations were found between Emotional Regulation and the BSI Depression factor, P < .0001, as well as the BSI Anxiety factor, P < .0001. Kappa coefficients indicated the highest agreement between the Emotional Regulation factor on the CCSS-NCQ and all factors on the BSI. Task Efficiency and the Memory factors from the CCSS-NCQ were also correlated with the BSI factors.
Discriminative validity was examined by comparing the neurocognitive factors between cancer survivors who are at “high risk” versus “low risk” for neurocognitive dysfunction. As described above, a “Neurologic” group and a “Frontal CRT” group were compared with a group of “Healthy” survivors and a sibling group. Multivariate analysis was done to assess group differences. As presented in Table 4, after controlling for age and sex, there were significant differences between the 4 groups on the scores for Task Efficiency, Emotional Regulation, Organization, and Memory. For all factors, both the Neurologic and Frontal CRT groups displayed more overall symptoms than the Healthy survivors group. No significant differences were identified between the 2 “high-risk” groups, or between the Healthy survivors and the siblings. Although the Frontal CRT group displayed more symptoms than the Healthy survivors, this was not simply a function of diagnosis with a CNS tumor. The entire CCSS sample included 801 patients with CNS tumors, whose performance spanned the entire range of possible scores on Task Efficiency (range, 9-27), Emotional Regulation (range, 3-9), Organization (range, 3-9), and Memory (range, 4-12). Thus, some CNS tumor patients reported no impairment, whereas others appeared significantly impaired.
|Group 1 Frontal CRT Mean (SD)||Group 2 Neurologic Mean (SD)||Group 3 Healthy Mean (SD)||Group 4 Sibling Mean (SD)||F||Group Comparison*|
|Task efficiency||16.5 (5.18)||16.2 (5.17)||11.8 (3.26)||11.9 (3.12)||127.65||1,2>3,4|
|Emotional regulation||5.5 (1.81)||5.7 (1.81)||5.1 (1.67)||5.0 (1.60)||11.82||1,2>3,4|
|Organization||4.9 (1.77)||5.0 (1.80)||4.4 (1.50)||4.6 (1.61)||12.03||2>3,4 1>3|
|Memory||7.6 (2.47)||7.1 (2.36)||5.8 (1.85)||5.8 (1.77)||71.41||1,2>3,4|
As a method to demonstrate clinical significance, ratings were reclassified based on the frequency of occurrence of an item being rated as “Often a Problem” (ie, the highest possible rating available on the CCSS-NCQ). Logistic regression was used to calculate ORs for the Healthy survivors group, the Neurologic group, and the Frontal CRT group compared with the sibling group. Table 5 presents the result of this analysis. Compared with siblings, the “high-risk” groups demonstrated significantly increased rates of problem ratings for each of the 4 CCSS-NCQ factors, particularly for Task Efficiency and Memory. Equally important to note is the finding that the “low-risk” group of Healthy survivors demonstrated no significant increase in likelihood of problems compared with the sibling sample.
|Group||Task Efficiency OR (95% CI)||Emotional Regulation OR (95% CI)||Organization OR (95% CI)||Memory OR (95% CI)|
|Healthy survivors||1.0 (0.7-1.4)||1.4 (1.0-2.0)||0.9 (0.6-1.3)||1.4 (0.9-2.0)|
|Neurologic survivors||6.8 (4.5-10.2)||2.7 (1.8-4.1)||2.0 (1.3-3.1)||4.6 (3.0-7.2)|
|Frontal CRT survivors||7.0 (4.8-10.2)||2.5 (1.7-3.7)||2.1 (1.4-3.1)||7.0 (4.7-10.5)|
In this study, we examined the validity of a brief self-report questionnaire in the assessment of neurocognitive functioning in long-term adult survivors of childhood cancer. This new questionnaire was designed to expand upon available questionnaire options by including items hypothesized to be sensitive to the effects of cancer therapy. Although current options tend to assess executive functions or behavioral symptoms, they generally do not focus on assessment of attention skills, processing speed, or memory and learning functions. Thus, for the new questionnaire, items addressing these specific skills were added to contextually similar items currently used to assess executive functions. The resultant CCSS-NCQ identified 4 factors that assess attention and processing speed (ie, “Task Efficiency”), emotional reactivity and frustration tolerance (ie, “Emotional Regulation”), organization of one's environment (ie, “Organization”), and working and long-term memory (ie, “Memory”). These 4 factors demonstrated good internal consistency, with a consistent pattern in both adult survivors of childhood cancer and their adult siblings. In addition, the procedure used with the development of this new measure ensured the control for demographic variables that are often relevant to cognitive functioning. That is, whereas the currently available measures of executive functions are standardized on a reference sample equated to the US Census, the CCSS-NCQ is referenced to siblings with a similar educational level and background. For example, the reference sample for the BRIEF-A includes 25.8% college graduates, which is consistent with the 24.7% in the 2002 US Census.16 However, this is significantly less than the 46.6% college graduation rate in the current CCSS sample. As cognitive abilities are impacted by educational experiences, using a reference group with lower educational attainment may result in underestimation in the rate of symptoms of mild cognitive dysfunction.
The factors derived from this new questionnaire correspond to established neurocognitive constructs. Emotional regulation, organization, and working memory are classically identified as neurocognitive abilities that fall within the category of “executive functions.”17 Attention, processing speed and long-term memory are important neurocognitive skills related to more basic levels of information processing.18 Task Efficiency correlated well with attention deficits, as evident on the BPI, even when reported by different raters across a span of many years. This finding may suggest the presence of a persistent and pervasive pattern of symptoms that is apparent to both the survivor and significant others. The Emotional Regulation factor also correlates well with the BSI, an established measure of emotional symptomatology. Although the CCSS-NCQ and BSI factors appear to address different aspects of emotional symptoms, with the BSI focusing on the emotional experience and the CCSS-NCQ focusing on emotional control, they are related, as would be expected, because experience is a necessary precursor to control. Of note, ratings on the BSI were also moderately correlated with the Task Efficiency and Memory factors from the CCSS-NCQ. This is not unexpected, as current literature demonstrates a relationship between active emotional symptoms and symptoms of attention and memory problems.51‒54 That is, emotional state can interfere with attention to and retention of new information.
The CCSS-NCQ demonstrated excellent discriminative validity. Ratings on the factors discriminated between healthy survivors and survivors at risk for neurocognitive impairment. Survivors with a history of an identified neurological event (ie, stroke or epilepsy) reported significantly more symptoms than healthy survivors. Similarly, survivors with a history of intense cranial radiation therapy (ie, ≥35 Gy) to the frontal area of their brains reported significantly more symptoms than healthy survivors, although they did not significantly differ from survivors with neurological events. Although the ORs for frequently occurring problem behaviors were significantly higher in these at-risk groups across all 4 CCSS-NCQ factors, the Task Efficiency and Memory factors appeared particularly impacted by an increased risk level. This finding is consistent with the literature that indicates problems with attention, processing speed, and memory as being a pattern commonly observed in long-term cancer survivors who have a history of CNS disease and/or CNS treatment.4, 5, 20 In fact, in a recent meta-analysis of 28 empirical studies, the average reported effect size in survivors of childhood leukemia was slightly higher for measures of attention and processing speed than was the effect size for measures of executive functions.5 The combination of this difference between risk groups with the lack of a significant difference between healthy survivors and siblings may suggest that the symptoms reported on the CCSS-NCQ are specific to CSN disease and/or CNS treatment rather than the general cancer experience.
The initial findings of this investigation suggest that the CCSS-NCQ is a reliable and valid method for monitoring neurocognitive outcomes in adult survivors of pediatric cancer. Such monitoring is recommended following treatment with cranial irradiation and/or antimetabolite chemotherapy.6 Ideally, all such individuals would undergo routine neurocognitive evaluations. However, as such evaluations are difficult to obtain, the CCSS-NCQ may offer a practical and efficient alternative for research purposes.
A limitation of the current study is the inability to examine the relationship between the CCSS-NCQ and performance-based assessment using standardized direct measurement of neurocognitive functioning. Direct performance measures are the “gold standard” for assessing neurocognitive functioning in participants. Having such a standard in the CCSS cohort would provide for calculation of enhanced sensitivity and specificity data for the CCSS-NCQ. An additional limitation is that data are not available to examine change in CCSS-NCQ ratings over time. Research is currently underway with a smaller subset of these patients at St. Jude Children's Research Hospital to obtain direct neurocognitive performance measures and to reassess performance using the CCSS-NCQ. These new data will provide more precise indices of sensitivity and specificity, as well as test-retest reliability. A final limitation is related to the type of cancer treatment used in the CCSS cohort. Because this cohort was diagnosed and treated between 1970 and 1986, treatment agents related to neurocognitive dysfunction were primarily CRT and IT methotrexate. High-dose IV methotrexate and dexamethasone, 2 currently used chemotherapy agents associated with neurocognitive dysfunction, were not often used at that time.55, 56
- 1SEER Cancer Statistics Review, 1975‒2004. Bethesda, MD: National Cancer Institute; 2007., , , et al.
- 7Manual for the ASEBA School-Age Forms and Profiles. Burlington, VT: ASEBA; 2001., .
- 8Behavior Assessment System for Children-Manual. Circle Pine, MN: American Guidance Service, Inc; 1992., .
- 9Brief Symptom Inventory (BSI): Administration, Scoring, and Procedures Manual. Minneapolis, MN: NCS Pearson; 2000..
- 10MMPI-2: Manual for Administration and Scoring. Minneapolis: University of Minnesota Press; 1989., , , , .
- 13Neurobehavioral Functioning Inventory. San Antonio, TX: The Psychological Corporation; 1999., , .
- 14Cognitive Behavior Rating Scales Research Edition. Lutz, FL: Psychological Assessment Resources, Inc; 1987..
- 15Frontal Systems Behavior Scale. Lutz, FL: Psychological Assessment Resources, Inc; 2001., .
- 16Behavior Rating Inventory of Executive Function—Adult Version. Lutz, FL: Psychological Assessment Resources, Inc; 2005., , .
- 17Higher-order cognitive impairments and frontal lobe lesions in man. In: LevinHS,BentonAL, eds. Frontal Lobe Function and Dysfunction. New York, NY: Oxford University Press; 1991., .
- 18Neuropsychological Assessment. 4th ed. New York, NY: Oxford University Press; 2004; 125–138., , .
- 19Behavior Rating Inventory of Executive Function. Odessa, FL: Psychological Assessment Resources, Inc; 2000., , , .
- 25Behavior Problems Index. Washington, DC: Child Trends Inc; 1986., .
- 26Manual for the Child Behavior Checklist. Burlington: University of Vermont, Department of Psychiatry; 1991..
- 29Coefficient alpha and the internal structure of tests. Psychometrika. 1951; 16: 297‒334..
- 31Greater attention problems during childhood predict poorer executive functioning in late adolescence. Psychol Sci. 2007; 18: 893‒900., , , et al.Direct Link: