• Childhood Cancer Survivor Study;
  • neurocognitive;
  • questionnaire;
  • late effects


  1. Top of page
  2. Abstract


Up to 40% of childhood cancer survivors may experience neurocognitive impairment in 1 or more specific domains. As such, regular monitoring has been recommended for patients exposed to cranial irradiation and/or antimetabolite chemotherapy. This study reports the results of a questionnaire developed to identify those survivors who may be experiencing neurocognitive problems.


Participants for this study were 7121 members of the Childhood Cancer Survivor Study cohort (6739 survivors and 382 siblings). These participants completed a new neurocognitive questionnaire designed to assess functions commonly affected by cancer therapy, as well as a standard measure of emotional functioning. A measure of cognitive and emotional functioning was also completed on a subset of the patients roughly 7 years before the current questionnaire. Responses to the questionnaires among subgroups of survivors were then analyzed to examine the reliability and validity of the new neurocognitive questionnaire.


Four reliable factors were identified that assessed task efficiency, emotional regulation, organization, and memory skills. These neurocognitive factors accurately discriminated survivors who were at “high risk” for neurocognitive dysfunction, because of neurologic abnormalities or a history of intensive focal cranial irradiation, from healthy “low-risk” survivors and siblings.


The questionnaire demonstrated excellent reliability, as well as construct and discriminative validity. It appears to be a practical and efficient tool for monitoring neurocognitive outcomes in adult survivors of pediatric cancer. Cancer 2008. © 2008 American Cancer Society.

Today in the US, approximately 80% of the nearly 12,000 children and adolescents diagnosed with cancer each year can expect to survive at least 5 years.1 With increased survival, attention has turned to quality of life and functional outcome after treatment, including neurocognitive performance. Studies suggest up to 40% of childhood cancer survivors may experience neurocognitive impairment in 1 or more specific domains (eg, processing speed, attention, memory).2‒5 Impairment in these domains impede the learning of new information, along with maintenance of previously learned information, and ultimately lead to declines in global intellect. This in turn results in poor academic and vocational success, low self-esteem, and behavioral or emotional disorders.

Regular monitoring of neurocognitive outcomes has been recommended for patients exposed to cranial irradiation and/or antimetabolite chemotherapy during treatment of pediatric cancer.6 Ideally, such monitoring would involve routine comprehensive neurocognitive evaluations. However, this approach may not be practical for a variety of reasons, including excessive cost, required time, and limited professional resources. This study reports the results of a questionnaire developed to identifying those survivors who may be experiencing neurocognitive problems.

Questionnaires are an established method for screening a variety of functions. Such scales have been used for many years in the assessment of emotional and behavioral problems in children (eg, the Child Behavior Checklist, Behavior Assessment System for Children), as well as adults (eg, Minn Multiphasic Personality Inventory, Brief Symptom Inventory).7‒10 The assessment of children typically involves completion of ratings by parents/guardians, whereas it is customary to collect self-reports for adults. More recently, the use of rating scales has expanded to aid in the assessment of health-related behaviors or outcomes, such as sleep habits,11 quality of life,12 and neurocognitive function.13, 14 A recent extension of neurocognitive ratings has been the focus on specific executive functions.15, 16

Currently, there are 2 scales designed to assess general neurocognitive functioning in adults. The Cognitive Behavior Rating Scales (CBRS) was standardized in adults aged 30 to 89 years, and generates measures of language deficits, agitation, need for routine, depression, higher cognitive deficits, memory, dementia, apraxia, and disorientation.14 The CBRS focuses on significant impairment and is generally used in the assessment of individuals with dementia. The Neurobehavioral Functioning Inventory (NFI) was standardized on adults aged 17 to 80 years, and generates indices of depression, somatic complaints, memory/attention problems, communication difficulties, aggression, and motor deficits.13 The NFI is typically used in rehabilitation settings for treatment planning and tracking change over time.

There are 2 widely used scales designed to evaluate executive functions in adults. Executive functions are generally defined as abilities required for the planning, implementing, monitoring, and adaptation of behavior for success in one's environment.17, 18 The Behavior Rating Inventory of Executive Function (BRIEF) is a widely used questionnaire designed to assess real-world aspects of executive dysfunction in children.19 The recently released adult version of the BRIEF (BRIEF-A) includes 2 scales: Behavioral Regulation Index (BRI) and Metacognition Index (MI). BRI is comprised of 4 subscales (Inhibit, Shift, Emotional Control, and Self-Monitor), and the MI is comprised of 5 subscales (Initiate, Working Memory, Plan/Organize, Task Monitor, and Organization of Materials).16 The Frontal Systems Behavior Scale (FrSBe) was originally developed for use in adults and includes 3 major scales: Apathy, Disinhibition, and Executive Dysfunction.15

The utility of these scales in the assessment of functioning in long-term survivors of childhood cancer is uncertain. When neurocognitive deficits do occur in long-term survivors, they typically involve problems with attention, processing speed, and/or memory.4, 5, 20 The CBRS and NFI have demonstrated sensitivity to general dysfunction and emotional adjustment in dementia and after acute brain injury.14, 21 However, neither of these scales differentiate between specific neurocognitive skills. The BRIEF-A and FrSBe focus on assessment of executive dysfunction, which may indeed be impacted in long-term survivors. However, neither measure includes assessment of attention, processing speed, or memory. Furthermore, the sensitivity and specificity of these scales have not been evaluated in adults who are long-term survivors of childhood cancer.

The European Organization for Research and Treatment of Cancer has developed a quality of life instrument that includes a measure of cognitive function.22 Although this scale does appear sensitive to the effects of a variety of cancers and therapies,23 the cognitive scale is 1-dimensional, and thus does not separate skills into neurocognitive subdomains that may demonstrate treatment specificity.

The purpose of the current study was to report the utility of a new rating scale developed for the ongoing Childhood Cancer Survivor Study (CCSS).24 This Neurocognitive Questionnaire (CCSS-NCQ) was constructed to assess skills that are reported to be sensitive to the effects of radiation and antimetabolite chemotherapy in long-term cancer survivors, as well as skills included in currently established measures of executive functions.


  1. Top of page
  2. Abstract


Participants for these analyses were members of the CCSS cohort. The CCSS is a retrospective cohort study designed to evaluate the impact of childhood cancer and its treatment on long-term function and health. Details of the criteria for cohort inclusion have been reported previously.24 Briefly, eligible participants were treated for 1 of 8 childhood cancer diagnoses at 26 institutions between 1970 and 1986 when younger than 21 years of age. Cohort entry was limited to those individuals who survived for at least 5 years after their original diagnosis. The human subjects committee at each of the collaborating institutions approved the study protocol before participant enrollment. Participants provided informed consent for the questionnaires and separate consent for medical record abstraction. Medical record abstraction was completed for all consenting participants. Study participants completed a 24-page baseline questionnaire between 1995 and 1996. This questionnaire surveyed sociodemographic information, medical history, and functional limitations. A second follow-up questionnaire, which included the CCSS-NCQ, was completed between 2002 and 2004 (the full survey questionnaire is available at; the CCSS-NCQ). Participants in the current analyses completed both the baseline questionnaire and the second follow-up survey, including the CCSS-NCQ. A total of 7121 participants from the original cohort completed the CCSS-NCQ. Diagnoses and demographics are presented in Table 1.

Table 1. Demographic Characteristics of Participants (Study Population for Exploratory Factor Analysis and Confirmatory Factor Analysis)
CharacteristicSurvivors (n=6739)Siblings (n=382)P
  1. CNS indicates central nervous system; NA, not applicable; SD, standard deviation.

Sex    .63
Race/ethnicity    <.01
Education    .22
 <High school graduate2513.892.4 
 High school graduate332049.718147.6 
 College graduate311546.619050.0 
 CNS tumor80111.9NA  
 Hodgkin lymphoma90813.5NA  
 Non-Hodgkin lymphoma5097.6NA  
 Wilms tumor6499.6NA  
 Soft tissue sarcoma6139.1NA  
 Bone cancer5958.8NA  
Age    <.01
 Age at interview32.17.634.18.4 
 Age at diagnosis8.55.9NA  


The CCSS-NCQ was developed by selecting a subset of items represented in an early investigational version of the BRIEF-A, to which additional items were added (the CCSS-NCQ was administered before the publication of the BRIEF-A). In conjunction with the primary author of the BRIEF,19 versions of items were selected to represent the following cognitive constructs: Organization of Materials, Working Memory, Plan/Organize, Initiate, Inhibit, and Emotional Control. To this item list, additional items were added that focused on processing speed, memory, and academic functioning. This resulted in the 25-item CCSS-NCQ. The items were presented with instructions asking participants to report the degree to which they experienced any of the 25 problems over the past 6 months. Possible responses were presented using a Likert scale ranging from 1 (“Never a Problem”) to 3 (“Often a Problem”). Of the participants who completed the CCSS-NCQ, 6739 were survivors, and 382 were siblings.

The Behavior Problem Index (BPI)25 was originally developed for the National Health Survey by taking a subset of questions from the Child Behavior Checklist.26 These questions were completed by parents of adolescents 12-17 years of age during the baseline questionnaire (ie, 1995-1996). For each of the 32 items, parents were asked to describe their child's behavioral and emotional functioning. Possible responses were presented using a Likert scale ranging from 1 (“Not True”) to 3 (“Often True”). The BPI has recently been validated in a CCSS sample.27 In that report, the following 5 factors were identified: Depression/Anxiety, Headstrong Behavior, Attention Deficit, Peer Conflict, and Antisocial behavior. Of those participants who completed the CCSS-NCQ, 1671 parents had completed the BPI during the baseline questionnaire.

The Brief Symptom Inventory-18 (BSI) is an 18-item screening questionnaire designed to assess symptoms of depression, anxiety, and somatic complaints.9 Participants were asked to describe the extent to which they have been distressed or bothered by each symptom in the previous 7 days. Possible responses were presented using a Likert scale ranging from 1 (“Not at All”) to 5 (“Extremely”). This measure was collected in the CCSS follow-up survey at the same time the CCSS-NCQ was obtained. As with the latter scale, 6739 survivors and 382 siblings completed the BSI. A recent report using the CCSS sample has supported the initial factor structure of the BSI in adult survivors of childhood cancer: Depression, Anxiety, and Somatic Complaints.28

To recap, the BPI was administered once during the baseline survey conducted between 1995-1996, whereas the BSI and CCSS-NCQ were each administered once during a follow-up survey conducted between 2002 and 2004.

Statistical Analyses

An exploratory factor analysis was conducted using all reports collected from siblings who completed the 25-item CCSS-NCQ. The maximum likelihood method was used to extract the factors, which were then subjected to a promax (oblique) rotation. Internal consistency of the factors was assessed using Cronbach Alpha.29 A confirmatory factor analysis was then conducted using all reports from cancer survivors. The weighted least squares method was used, as the multivariate normal assumption was in doubt.

Construct validity of the factors was explored by comparing performance on the CCSS-NCQ factors to the BPI items collected on the CCSS baseline questionnaire. Although the BPI was collected roughly 7 years before the CCSS-NCQ, the BPI was administered during adolescence, a period of development when cognitive patterns become engrained. When attention problems persist into adolescence, they typically continue into adulthood and expand to impact executive functioning.30, 31 Thus, parent report of significant problems during adolescent should be reflected in self-reports, even 7 years later. Concurrent validity was examined by comparing the established factor scores to the factors from the BSI, which was also collected during the second follow-up questionnaire. Discriminative validity was examined by comparing the CCSS-NCQ factors between cancer survivors at “high risk” to those at “low risk” for neurocognitive dysfunction. For this purpose 2 “high-risk” groups were defined: 1 group (n = 172) included all survivors with epilepsy and/or cerebrovascular abnormalities (ie, Neurologic group); the other group (n = 247) included survivors treated with high-dose (≥35 Gray [Gy]) cranial radiation therapy (CRT) to the frontal area of their brains (ie, Frontal CRT group). A cutoff of ≥35-Gy CRT was established based on the attempt to identify a “high-risk” group for validation of the instrument. In a recent review of CRT effects on late neurocognitive sequelae, >35 Gy was a level consistently associated with significant impairment.32 Although lower doses of CRT and chemotherapy treatment (eg, 18 Gy CRT + intrathecal [IT] and/or high dose intravenous [IV] methotrexate) present elevated risk, the risk is not consistently perceived to be “high.” Furthermore, at least 2 recent reports question the impact of lower doses of CRT and chemotherapy on neurocognitive functions.33, 34 Neurocognitive dysfunction has also been consistently reported in samples of adults with epilepsy,35‒39 stroke,40‒42 and cerebrovascular abnormalities.43‒45 Increased incidence of such neurologic disorders has been documented in survivors of childhood cancer treated with cranial radiation.46, 47 These “high-risk” groups were compared with a “low-risk” group of healthy survivors who had no history of central nervous system (CNS) disease, no history of CNS treatment, and no history of major organ complication or procedures (n = 688). These groups were also compared with the sibling group (n = 382). Multivariate analysis was done to assess group differences. As a method to evaluate clinical significance of the ratings, performance on each scale was reclassified based on the frequency of an item being rated as “Often a Problem” (ie, the highest possible rating available). The frequency of these ratings were then calculated for the sibling group, and logistic regression was used to evaluate odds ratios (ORs) for the likelihood of higher scores in each scale for each of the “high-risk” and “low-risk” survivor groups compared with the siblings. Robust standard errors were calculated based on generalized estimating equation methods, to account for intrafamily correlation between survivors and siblings, and were used to obtain 95% confidence intervals. All regression models were adjusted for age and sex. Two-sided P values <.05 were considered significant.


  1. Top of page
  2. Abstract

During the exploratory factor analysis (EFA) in 382 siblings, 4 factors met criteria for retention based on eigenvalues, assessment of the scree plot, and face validity of grouped items. In interpreting the rotated factor pattern, an item was said to load on a given factor if the factor loading was 0.40 or greater. The factor loadings from the exploratory analysis and the internal consistency estimates of the rotated factors are presented in Table 2. Four factors representing 19 of the 25 variables were obtained: Task Efficiency, Emotional Regulation, Organization, and Memory. The Task Efficiency factor accounted for the major proportion of variance and was comprised of items related to initiating and completing tasks in a rapid manner. Emotional Regulation significantly contributed to the overall variance and was comprised of items related to emotional reactivity and frustration tolerance. The Organization factor was comprised of items related to organization of the individual's environment. The fourth factor, Memory, was comprised of items related to both working memory and long-term memory (ie, recall of information learned previously).

Table 2. Questionnaire Items and Corresponding Factor Loadings From the Rotated Factor Pattern Matrix
QuestionTask EfficiencyEmotional ToleranceOrganizationMemory
  • *

    Values >0.40.

  • Questions that did not significantly contribute to a factor.

  • Cronbach alpha.27

Gets upset easily*−0.070.760.00−0.13
Takes longer to complete work*0.640.050.02−0.06
Does not think of consequences0.
Forgets instructions easily*0.20−
Problems completing my work*0.650.010.21−0.14
Difficulty recalling things learned before*
Gets frustrated easily*−0.060.74−0.020.15
Mood changes frequently*0.090.69−0.04−0.02
Different ways to solve a problem0.240.33−0.060.13
Forgot what doing in middle of things*−
Problems with self-motivation*0.410.030.210.10
Trouble finding things in bedroom*−
Easily overwhelmed*0.470.38−0.040.04
Trouble with multitasking*0.470.130.070.05
Blurts things out0.
Desk/workspace a mess*−0.01−0.070.720.02
Trouble remembering things0.060.03−0.030.77
Trouble prioritizing activities*0.420.060.260.05
Reads slowly*0.51−0.13−0.080.12
Slower than others*0.73−0.12−0.080.05
Does not work well under pressure*0.540.10−0.100.09
Trouble solving math problems0.320.01−0.180.29
Internal consistency0.840.770.780.78

Confirmatory analysis (CFA) was conducted in the survivor population (n = 6739) using the 4-factor solution from the EFA. The goodness of fit index (GFI) was 0.91, and the root mean square error of approximation (RMSEA) was 0.05. RMSEA is an index of fit that is less influenced by sample size, with recommended levels being 0.05 or below.48 These indices suggest a reasonable fit of the 4-factor structure to the cancer survivor sample. A CFA was also conducted using only those survivors determined to be at “high risk” for neurocognitive dysfunction (ie, the Neurologic group and the Frontal CRT group). The fit index for this at-risk group was acceptable (GFI = 0.93). As part of the consideration of other factor solutions in explaining the pattern of data from the survivors, additional CFAs were conducted. CFA was used with a single factor to determine whether the combined response to all 25 items best explained the variability in survivor responding. This resulted in a significantly reduced fit index (GFI = 0.83). A CFA was also conducted using a 2-factor solution based on the factors from which the represented items on the BRIEF-A were selected (ie, Behavioral Regulation and Metacognition). Again, this resulted in a significantly reduced fit index (GFI = 0.88). Thus, the 4-factor solution produced the best fit during confirmation.

Construct validity of the 4 CCSS-NCQ factors was demonstrated through correlation with the BPI factors. Of the 4 factors derived, Task Efficiency was expected to correlate most highly with the BPI factor Attention Deficit. This was, in fact, the resultant outcome. Table 3 presents correlations between the factors on the 2 rating scales, along with kappa coefficients (κ). Kappa is an index of agreement between dichotomous variables, which attempts to account for base rates in the population, with “fair agreement” being a coefficient ≥0.20.49, 50 The relevance of these correlations and agreement coefficients is increased through recognition that the questionnaires were completed by 2 different raters (ie, cohort completed the CCSS-NCQ, whereas parents completed the BPI) over an interval of roughly 7 years (ie, the BPI was completed between 1995 and 1996, and the CCSS-NCQ was completed between 2002 and 2004). Of note, the Depression/Anxiety factor from the BPI was correlated more strongly with the CCSS-NCQ Emotional Regulation than any of the other NCQ factors. Overall, the pattern of correlations with the BPI provides initial support for the 4-factor constructs associated with the CCSS-NCQ.

Table 3. Pearson Correlation (r)* and Cohen Kappa (κ) Coefficients for Childhood Cancer Survivor Study Neurocognitive Questionnaire, Behavior Problem Index, and Brief Symptom Inventory-18 Factors
 Task Efficiency, r (κ)Emotional Regulation, r (κ)Organization, r (κ)Memory r (κ)
  • Kappa coefficients ≥0.20 indicate at least fair agreement between dichotomous variables.42, 43

  • *

    Given the large sample sizes, all P values are <.0001.

Behavior Problem Index    
 Depression/Anxiety0.25 (0.13)0.27 (0.19)0.12 (0.08)0.20 (0.10)
 Headstrong Behavior0.24 (0.10)0.30 (0.16)0.15 (0.07)0.21 (0.10)
 Social Deviance0.22 (0.11)0.27 (0.17)0.18 (0.10)0.20 (0.14)
 Attention Deficit0.40 (0.21)0.32 (0.18)0.25 (0.14)0.32 (0.19)
 Peer Conflict0.32 (0.16)0.25 (0.16)0.17 (0.07)0.26 (0.14)
Brief Symptom Inventory    
 Depression0.44 (0.27)0.49 (0.34)0.23 (0.17)0.35 (0.23)
 Somatization0.35 (0.20)0.38 (0.24)0.20 (0.13)0.34 (0.22)
 Anxiety0.38 (0.22)0.49 (0.32)0.23 (0.12)0.34 (0.19)

Concurrent validity was assessed through comparison of responses to the CCSS-NCQ and the BSI, both of which were completed during the same assessment. Here, it was expected that the CCSS-NCQ Emotional Regulation factor would correlate more highly with the BSI factors than would the other CCSS-NCQ factors. Again, as evident in Table 3, this was the resultant outcome. Significant correlations were found between Emotional Regulation and the BSI Depression factor, P < .0001, as well as the BSI Anxiety factor, P < .0001. Kappa coefficients indicated the highest agreement between the Emotional Regulation factor on the CCSS-NCQ and all factors on the BSI. Task Efficiency and the Memory factors from the CCSS-NCQ were also correlated with the BSI factors.

Discriminative validity was examined by comparing the neurocognitive factors between cancer survivors who are at “high risk” versus “low risk” for neurocognitive dysfunction. As described above, a “Neurologic” group and a “Frontal CRT” group were compared with a group of “Healthy” survivors and a sibling group. Multivariate analysis was done to assess group differences. As presented in Table 4, after controlling for age and sex, there were significant differences between the 4 groups on the scores for Task Efficiency, Emotional Regulation, Organization, and Memory. For all factors, both the Neurologic and Frontal CRT groups displayed more overall symptoms than the Healthy survivors group. No significant differences were identified between the 2 “high-risk” groups, or between the Healthy survivors and the siblings. Although the Frontal CRT group displayed more symptoms than the Healthy survivors, this was not simply a function of diagnosis with a CNS tumor. The entire CCSS sample included 801 patients with CNS tumors, whose performance spanned the entire range of possible scores on Task Efficiency (range, 9-27), Emotional Regulation (range, 3-9), Organization (range, 3-9), and Memory (range, 4-12). Thus, some CNS tumor patients reported no impairment, whereas others appeared significantly impaired.

Table 4. Mean Childhood Cancer Survivor Study Neurocognitive Questionnaire Scores for Various Survivor Risk Groups
 Group 1 Frontal CRT Mean (SD)Group 2 Neurologic Mean (SD)Group 3 Healthy Mean (SD)Group 4 Sibling Mean (SD)FGroup Comparison*
  • Adjusted for age and sex through covariate analysis.

  • CRT indicates cranial radiation therapy; SD, standard deviation.

  • *

    Significant differences based on P < .05 with Bonferroni correction for multiple comparisons.

Task efficiency16.5 (5.18)16.2 (5.17)11.8 (3.26)11.9 (3.12)127.651,2>3,4
Emotional regulation5.5 (1.81)5.7 (1.81)5.1 (1.67)5.0 (1.60)11.821,2>3,4
Organization4.9 (1.77)5.0 (1.80)4.4 (1.50)4.6 (1.61)12.032>3,4 1>3
Memory7.6 (2.47)7.1 (2.36)5.8 (1.85)5.8 (1.77)71.411,2>3,4

As a method to demonstrate clinical significance, ratings were reclassified based on the frequency of occurrence of an item being rated as “Often a Problem” (ie, the highest possible rating available on the CCSS-NCQ). Logistic regression was used to calculate ORs for the Healthy survivors group, the Neurologic group, and the Frontal CRT group compared with the sibling group. Table 5 presents the result of this analysis. Compared with siblings, the “high-risk” groups demonstrated significantly increased rates of problem ratings for each of the 4 CCSS-NCQ factors, particularly for Task Efficiency and Memory. Equally important to note is the finding that the “low-risk” group of Healthy survivors demonstrated no significant increase in likelihood of problems compared with the sibling sample.

Table 5. Odds Ratios for “Often a Problem” Rating
GroupTask Efficiency OR (95% CI)Emotional Regulation OR (95% CI)Organization OR (95% CI)Memory OR (95% CI)
  1. Adjusted for age and sex through covariate analysis.

  2. OR indicates odds ratio; CI, confidence interval; CRT, cranial radiation therapy.

Siblings (Ref)
Healthy survivors1.0 (0.7-1.4)1.4 (1.0-2.0)0.9 (0.6-1.3)1.4 (0.9-2.0)
Neurologic survivors6.8 (4.5-10.2)2.7 (1.8-4.1)2.0 (1.3-3.1)4.6 (3.0-7.2)
Frontal CRT survivors7.0 (4.8-10.2)2.5 (1.7-3.7)2.1 (1.4-3.1)7.0 (4.7-10.5)


  1. Top of page
  2. Abstract

In this study, we examined the validity of a brief self-report questionnaire in the assessment of neurocognitive functioning in long-term adult survivors of childhood cancer. This new questionnaire was designed to expand upon available questionnaire options by including items hypothesized to be sensitive to the effects of cancer therapy. Although current options tend to assess executive functions or behavioral symptoms, they generally do not focus on assessment of attention skills, processing speed, or memory and learning functions. Thus, for the new questionnaire, items addressing these specific skills were added to contextually similar items currently used to assess executive functions. The resultant CCSS-NCQ identified 4 factors that assess attention and processing speed (ie, “Task Efficiency”), emotional reactivity and frustration tolerance (ie, “Emotional Regulation”), organization of one's environment (ie, “Organization”), and working and long-term memory (ie, “Memory”). These 4 factors demonstrated good internal consistency, with a consistent pattern in both adult survivors of childhood cancer and their adult siblings. In addition, the procedure used with the development of this new measure ensured the control for demographic variables that are often relevant to cognitive functioning. That is, whereas the currently available measures of executive functions are standardized on a reference sample equated to the US Census, the CCSS-NCQ is referenced to siblings with a similar educational level and background. For example, the reference sample for the BRIEF-A includes 25.8% college graduates, which is consistent with the 24.7% in the 2002 US Census.16 However, this is significantly less than the 46.6% college graduation rate in the current CCSS sample. As cognitive abilities are impacted by educational experiences, using a reference group with lower educational attainment may result in underestimation in the rate of symptoms of mild cognitive dysfunction.

The factors derived from this new questionnaire correspond to established neurocognitive constructs. Emotional regulation, organization, and working memory are classically identified as neurocognitive abilities that fall within the category of “executive functions.”17 Attention, processing speed and long-term memory are important neurocognitive skills related to more basic levels of information processing.18 Task Efficiency correlated well with attention deficits, as evident on the BPI, even when reported by different raters across a span of many years. This finding may suggest the presence of a persistent and pervasive pattern of symptoms that is apparent to both the survivor and significant others. The Emotional Regulation factor also correlates well with the BSI, an established measure of emotional symptomatology. Although the CCSS-NCQ and BSI factors appear to address different aspects of emotional symptoms, with the BSI focusing on the emotional experience and the CCSS-NCQ focusing on emotional control, they are related, as would be expected, because experience is a necessary precursor to control. Of note, ratings on the BSI were also moderately correlated with the Task Efficiency and Memory factors from the CCSS-NCQ. This is not unexpected, as current literature demonstrates a relationship between active emotional symptoms and symptoms of attention and memory problems.51‒54 That is, emotional state can interfere with attention to and retention of new information.

The CCSS-NCQ demonstrated excellent discriminative validity. Ratings on the factors discriminated between healthy survivors and survivors at risk for neurocognitive impairment. Survivors with a history of an identified neurological event (ie, stroke or epilepsy) reported significantly more symptoms than healthy survivors. Similarly, survivors with a history of intense cranial radiation therapy (ie, ≥35 Gy) to the frontal area of their brains reported significantly more symptoms than healthy survivors, although they did not significantly differ from survivors with neurological events. Although the ORs for frequently occurring problem behaviors were significantly higher in these at-risk groups across all 4 CCSS-NCQ factors, the Task Efficiency and Memory factors appeared particularly impacted by an increased risk level. This finding is consistent with the literature that indicates problems with attention, processing speed, and memory as being a pattern commonly observed in long-term cancer survivors who have a history of CNS disease and/or CNS treatment.4, 5, 20 In fact, in a recent meta-analysis of 28 empirical studies, the average reported effect size in survivors of childhood leukemia was slightly higher for measures of attention and processing speed than was the effect size for measures of executive functions.5 The combination of this difference between risk groups with the lack of a significant difference between healthy survivors and siblings may suggest that the symptoms reported on the CCSS-NCQ are specific to CSN disease and/or CNS treatment rather than the general cancer experience.

The initial findings of this investigation suggest that the CCSS-NCQ is a reliable and valid method for monitoring neurocognitive outcomes in adult survivors of pediatric cancer. Such monitoring is recommended following treatment with cranial irradiation and/or antimetabolite chemotherapy.6 Ideally, all such individuals would undergo routine neurocognitive evaluations. However, as such evaluations are difficult to obtain, the CCSS-NCQ may offer a practical and efficient alternative for research purposes.

A limitation of the current study is the inability to examine the relationship between the CCSS-NCQ and performance-based assessment using standardized direct measurement of neurocognitive functioning. Direct performance measures are the “gold standard” for assessing neurocognitive functioning in participants. Having such a standard in the CCSS cohort would provide for calculation of enhanced sensitivity and specificity data for the CCSS-NCQ. An additional limitation is that data are not available to examine change in CCSS-NCQ ratings over time. Research is currently underway with a smaller subset of these patients at St. Jude Children's Research Hospital to obtain direct neurocognitive performance measures and to reassess performance using the CCSS-NCQ. These new data will provide more precise indices of sensitivity and specificity, as well as test-retest reliability. A final limitation is related to the type of cancer treatment used in the CCSS cohort. Because this cohort was diagnosed and treated between 1970 and 1986, treatment agents related to neurocognitive dysfunction were primarily CRT and IT methotrexate. High-dose IV methotrexate and dexamethasone, 2 currently used chemotherapy agents associated with neurocognitive dysfunction, were not often used at that time.55, 56


  1. Top of page
  2. Abstract
  • 1
    Ries LAG,Melbert D,Krapcho M, et al. SEER Cancer Statistics Review, 1975‒2004. Bethesda, MD: National Cancer Institute; 2007.
  • 2
    Moleski M. Neuropsychological, neuroanatomical, and neurophysiological consequences of CNS chemotherapy for acute lymphoblastic leukemia. Arch Clin Neuropsychol. 2000; 15: 603630.
  • 3
    Mulhern RK,Fairclough D,Ochs J. A prospective comparison of neuropsychologic performance of children surviving leukemia who received 18-Gy, 24-Gy, or no cranial irradiation. J Clin Oncol. 1991; 9: 13481356.
  • 4
    Butler RW,Haser JK. Neurocognitive effects of treatment for childhood cancer. Ment Retard Dev Disabil Res Rev. 2006; 12: 184191.
  • 5
    Campbell LK,Scaduto M,Sharp W, et al. A meta-analysis of the neurocognitive sequelae of treatment for childhood acute lymphocytic leukemia. Pediatr Blood Cancer. 2007; 49: 6573.
  • 6
    Nathan PC,Patel SK,Dilley K, et al. Guidelines for identification of, advocacy for, and intervention in neurocognitive problems in survivors of childhood cancer: a report from the Children's Oncology Group. Arch Pediatr Adolesc Med. 2007; 161: 798806.
  • 7
    Achenbach TM,Rescorla LA. Manual for the ASEBA School-Age Forms and Profiles. Burlington, VT: ASEBA; 2001.
  • 8
    Reynolds CR,Kamphaus RW. Behavior Assessment System for Children-Manual. Circle Pine, MN: American Guidance Service, Inc; 1992.
  • 9
    Derogatis LR. Brief Symptom Inventory (BSI): Administration, Scoring, and Procedures Manual. Minneapolis, MN: NCS Pearson; 2000.
  • 10
    Butcher JN,Dahlstrom WG,Graham JR,Tellegen AM,Kaemmer B. MMPI-2: Manual for Administration and Scoring. Minneapolis: University of Minnesota Press; 1989.
  • 11
    Buysse DJ,Reynolds CFIII,Monk TH,Berman SR,Kupfer DJ. The Pittsburgh Sleep Quality Index: a new instrument for psychiatric practice and research. Psychiatry Res. 1989; 28: 193213.
  • 12
    Varni JW,Seid M,Rode CA. The PedsQL: measurement model for the pediatric quality of life inventory. Med Care. 1999; 37: 126139.
  • 13
    Kreutzer JS,Seel RT,Marwitz JH. Neurobehavioral Functioning Inventory. San Antonio, TX: The Psychological Corporation; 1999.
  • 14
    Williams JM. Cognitive Behavior Rating Scales Research Edition. Lutz, FL: Psychological Assessment Resources, Inc; 1987.
  • 15
    Grace J,Malloy PF. Frontal Systems Behavior Scale. Lutz, FL: Psychological Assessment Resources, Inc; 2001.
  • 16
    Roth RM,Isquith PK,Gioia GA. Behavior Rating Inventory of Executive Function—Adult Version. Lutz, FL: Psychological Assessment Resources, Inc; 2005.
  • 17
    Shallice T,Burgess P. Higher-order cognitive impairments and frontal lobe lesions in man. In: LevinHS,BentonAL, eds. Frontal Lobe Function and Dysfunction. New York, NY: Oxford University Press; 1991.
  • 18
    Lezak MD,Howieson DB,Loring DW. Neuropsychological Assessment. 4th ed. New York, NY: Oxford University Press; 2004; 125138.
  • 19
    Gioia GA,Isquith PK,Guy SC,Kenworthy L. Behavior Rating Inventory of Executive Function. Odessa, FL: Psychological Assessment Resources, Inc; 2000.
  • 20
    Mulhern RK,Armstrong FD,Thompson SJ. Function-specific neuropsychological assessment. Med Pediatr Oncol. 1998;( suppl 1): 3440.
  • 21
    Kennedy RE,Livingston L,Riddick A,Marwitz JH,Kreutzer JS,Zasler ND. Evaluation of the Neurobehavioral Functioning Inventory as a depression screening tool after traumatic brain injury. J Head Trauma Rehabil. 2005; 20: 512526.
  • 22
    Aaronson NK,Ahmedzai S,Bergman B, et al. The European Organization for Research and Treatment of Cancer QLQ-C30: a quality-of-life instrument for use in international clinical trials in oncology. J Natl Cancer Inst. 1993; 85: 365376.
  • 23
    Coates A,Porzsolt F,Osoba D. Quality of life in oncology practice: prognostic value of EORTC QLQ-C30 scores in patients with advanced malignancy. Eur J Cancer. 1997; 33: 10251030.
  • 24
    Robison LL,Mertens AC,Boice JD, et al. Study design and cohort characteristics of the Childhood Cancer Survivor Study: a multi-institutional collaborative project. Med Pediatr Oncol. 2002; 38: 229239.
  • 25
    Zill N,Peterson J. Behavior Problems Index. Washington, DC: Child Trends Inc; 1986.
  • 26
    Achenbach TM. Manual for the Child Behavior Checklist. Burlington: University of Vermont, Department of Psychiatry; 1991.
  • 27
    Schultz KA,Ness KK,Whitton J, et al. Behavioral and social outcomes in adolescent survivors of childhood cancer: a report from the childhood cancer survivor study. J Clin Oncol. 2007; 25: 36493656.
  • 28
    Recklitis CJ,Parsons SK,Shih MC,Mertens A,Robison LL,Zeltzer L. Factor structure of the brief symptom inventory-18 in adult survivors of childhood cancer: results from the childhood cancer survivor study. Psychol Assess. 2006; 18: 2232.
  • 29
    Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951; 16: 297334.
  • 30
    Biederman J,Petty CR,Fried R, et al. Stability of executive function deficits into young adult years: a prospective longitudinal follow-up study of grown up males with ADHD. Acta Psychiatr Scand. 2007; 116: 129136.
  • 31
    Friedman NP,Haberstick BC,Willcutt EG, et al. Greater attention problems during childhood predict poorer executive functioning in late adolescence. Psychol Sci. 2007; 18: 893900.
    Direct Link:
  • 32
    Mulhern RK,Merchant TE,Gajjar A,Reddick WE,Kun LE. Late neurocognitive sequelae in survivors of brain tumours in childhood. Lancet Oncol. 2004; 5: 399408.
  • 33
    Spiegler BJ,Kennedy K,Maze R, et al. Comparison of long-term neurocognitive outcomes in young children with acute lymphoblastic leukemia treated with cranial radiation or high-dose or very high-dose intravenous methotrexate. J Clin Oncol. 2006; 24: 38583864.
  • 34
    Waber DP,De Moor C,Forbes PW, et al. The NIH MRI study of normal brain development: performance of a population based sample of healthy children aged 6 to 18 years on a neuropsychological battery. J Int Neuropsychol Soc. 2007; 13: 729746.
  • 35
    Kim CH,Lee SA,Yoo HJ,Kang JK,Lee JK. Executive performance on the Wisconsin Card Sorting Test in mesial temporal lobe epilepsy. Eur Neurol. 2007; 57: 3946.
  • 36
    Drane DL,Lee GP,Cech H, et al. Structured cueing on a semantic fluency task differentiates patients with temporal versus frontal lobe seizure onset. Epilepsy Behav. 2006; 9: 339344.
  • 37
    Locke DE,Berry DT,Fakhoury TA,Schmitt FA. Relationship of indicators of neuropathology, psychopathology, and effort to neuropsychological results in patients with epilepsy or psychogenic non-epileptic seizures. J Clin Exp Neuropsychol. 2006; 28: 325340.
  • 38
    McDonald CR,Delis DC,Norman MA,Tecoma ES,Iragui-Madozi VI. Is impairment in set-shifting specific to frontal-lobe dysfunction? Evidence from patients with frontal-lobe or temporal-lobe epilepsy. J Int Neuropsychol Soc. 2005; 11: 477481.
  • 39
    McDonald CR,Delis DC,Norman MA,Wetter SR,Tecoma ES,Iragui VJ. Response inhibition and set shifting in patients with frontal lobe epilepsy or temporal lobe epilepsy. Epilepsy Behav. 2005; 7: 438446.
  • 40
    Nys GM,van Zandvoort MJ,van der Worp HB,Kappelle LJ,de Haan EH. Neuropsychological and neuroanatomical correlates of perseverative responses in subacute stroke. Brain. 2006; 129(pt 8): 21482157.
  • 41
    Ziemus B,Baumann O,Luerding R, et al. Impaired working-memory after cerebellar infarcts paralleled by changes in BOLD signal of a cortico-cerebellar circuit. Neuropsychologia. 2007; 45: 20162024.
  • 42
    Stephens S,Kenny RA,Rowan E, et al. Association between mild vascular cognitive impairment and impaired activities of daily living in older stroke survivors without dementia. J Am Geriatr Soc. 2005; 53: 103107.
  • 43
    Komoda T,Drews T,Sakuraba S,Kubo M,Hetzer R. Executive cognitive dysfunction without stroke after long-term mechanical circulatory support. ASAIO J. 2005; 51: 764768.
  • 44
    Vicario A,Martinez CD,Baretto D,Diaz Casale A,Nicolosi L. Hypertension and cognitive decline: impact on executive function. J Clin Hypertens (Greenwich). 2005; 7: 598604.
  • 45
    Sachdev P. Homocysteine, cerebrovascular disease and brain atrophy. J Neurol Sci. 2004; 226: 2529.
  • 46
    Anderson V,Godber T,Smibert E,Ekert H. Neurobehavioural sequelae following cranial irradiation and chemotherapy in children: an analysis of risk factors. Pediatr Rehabil. 1997; 1: 6376.
  • 47
    Maddrey AM,Bergeron JA,Lombardo ER, et al. Neuropsychological performance and quality of life of 10 year survivors of childhood medulloblastoma. J Neurooncol. 2005; 72: 245253.
  • 48
    Floyd FJ,Widaman KF. Factor analysis in the development and refinement of clinical assessment instruments. Psychol Assess. 1995; 7: 286299.
  • 49
    Brennan P,Silman A. Statistical methods for assessing observer variability in clinical measures. BMJ. 1992; 304: 14911494.
  • 50
    Thompson WD,Walter SD. A reappraisal of the kappa coefficient. J Clin Epidemiol. 1988; 41: 949958.
  • 51
    Christopher G,MacDonald J. The impact of clinical depression on working memory. Cognit Neuropsychiatry. 2005; 10: 379399.
  • 52
    DeLuca AK,Lenze EJ,Mulsant BH, et al. Comorbid anxiety disorder in late life depression: association with memory decline over 4 years. Int J Geriatr Psychiatry. 2005; 20: 848854.
  • 53
    Frais AT. Depression and the causal role of specific memory system degenerations: link may be supported by reported therapeutic benefits of Omega 3 fatty acids. Med Hypotheses. 2007; 69: 6769.
  • 54
    Watts SE,Weems CF. Associations among selective attention, memory bias, cognitive errors and symptoms of anxiety in youth. J Abnorm Child Psychol. 2006; 34: 841852.
  • 55
    Reddick WE,Glass JO,Helton KJ, et al. Prevalence of leukoencephalopathy in children treated for acute lymphoblastic leukemia with high-dose methotrexate. AJNR Am J Neuroradiol. 2005; 26: 12631269.
  • 56
    Waber DP,Carpentieri SC,Klar N, et al. Cognitive sequelae in children treated for acute lymphoblastic leukemia with dexamethasone or prednisone. J Pediatr Hematol Oncol. 2000; 22: 206213.