Dr Joukje M. Oosterman PhD, Radboud University Nijmegen, Donders Institute for Brain, Cognition and Behaviour, Montessorilaan 3, 6500 HE Nijmegen, the Netherlands. Email: firstname.lastname@example.org
Background: Executive function deficits are commonly observed in many clinical populations, highlighting the importance of appropriate diagnostic tools to screen for these deficits. Most neuropsychological tests of executive function, however, are time-consuming and difficult to administer in the case of moderate to severe cognitive decline. The aim of the present study was to examine whether the Key Search Test, a short and easy to administer test, is a useful indicator of executive function deficits in a study sample with a diagnosis of cognitive impairment.
Methods: Participants consisted of elderly people visiting the memory clinic at the department of geriatrics of a university medical center (n= 140) and of elderly controls (n= 37). Next to the Key Search Test, other executive function tests and a memory test were administered.
Results: Low to moderate correlations were found between the Key Search Test and other executive function tests. Furthermore, although the Key Search Test discriminated significantly between intact and impaired executive function (AUC = 0.677, P < 0.001), sensitivity and specificity were low and no optimal cut-off point could be determined.
Conclusion: The Key Search test might not be an appropriate measure of executive functions in cognitively impaired individuals.
The worldwide ageing of the population in combination with the increasing prevalence of dementia necessitates the development and improvement of diagnostic tools for screening purposes. Although memory deficits are considered to be one of the core problems in these patient populations, an increasing number of studies point to the importance of screening for executive function deficits, as these deficits are commonly observed in most clinical populations. Notably, executive dysfunction might be a very strong predictor of future conversion to dementia in community-dwelling elderly.1 However, extensive neuropsychological testing is often difficult in older patients as a result of general factors, such as decreased motivation or a limited ability to concentrate. Similarly, the available executive function tests might not be easily administrable to patients with moderate to severe cognitive decline. For example, a previous study on normal and pathological ageing observed that, as a result of cognitive impairment, a substantial percentage of participants (nearly 12%) was unable to complete the Trail Making Test in accordance with the task instructions.2 The Trail Making Test is a frequently applied task of executive functions in the clinical setting,3 but the high drop-out rate shows that it might not be a useful indicator of executive function in pathological ageing. Also, the executive clock drawing task, another frequently applied task, was found to not be an appropriate indicator of executive function decline in elderly participants with subcortical ischaemic vascular disease.4 This urges the identification of a simple and easy to administer test of executive functions. The Key Search task is an ideal candidate for this purpose; it is easy to administer and administration time is usually short. This test is part of the Behavioural Assessment of the Dysexecutive Syndrome (BADS) battery that was designed as an ecologically valid measure of executive function.5 Participants are required to search for an imaginary key they have lost on a field (a square on a piece of paper). By drawing their search route, an indication of search strategy and planning ability can be deduced. Examples of intact and distorted search routes are presented in Figure 1. Several previous studies have successfully administered the Key Search Test to patient groups of varying aetiologies, including traumatic brain injury,6,7 dementia,8,9 schizophrenia,10 epilepsy,11 and Parkinson's disease.12 Performance on this test has been shown to correlate with other executive function measures, as well as more general intelligence indices.7 However, in that particular study, general intelligence was a stronger predictor of performance on the Key Search Test than executive ability. Furthermore, the participants included in that study had all suffered traumatic brain injury; hence, diagnostic accuracy of the Key Search Test for executive function deficits in people referred to memory clinics for the diagnostic work-up of suspected dementia is unclear.
The focus of the present study was to examine whether the Key Search Test can accurately indentify executive function deficits in a population referred to the memory clinic for cognitive complaints or suspected dementia. This clinical population includes people at risk of developing dementia or already in the early stages of dementia, which makes the identification of a screening instrument with high prognostic value (i.e. high sensitivity and specificity) for executive deficits in this population of crucial importance. We included a heterogeneous sample of patients, because the Key Search Test should be able to identify executive function deficits, regardless of the underlying diagnosis. Second, we examined the additional diagnostic value of this test to the Mini-Mental State Examination (MMSE)13 in classifying participants as either cognitively intact or cognitively impaired. The MMSE is a widely used screening instrument and has been shown to be a valid measure of cognitive decline in mild dementia, although it lacks a measure of executive function.14 As executive functions are affected in both normal ageing and the majority of the pathological ageing processes, such as Alzheimer's disease and vascular dementia, tests measuring these functions might have additional predictive value in detecting possible cognitive decline.
Test results of a total of 177 elderly people were used in the presented retrospective study. Of these participants, 140 were diagnosed at the memory clinic of the department of Geriatric Medicine of the Radboud University Nijmegen Medical Centre according to current international guidelines.15–18 All diagnoses were based on a thorough clinical interview and extensive neuropsychological and psychiatric screening. Accordingly, the following diagnoses were made: mild cognitive impairment (MCI; n= 65), Alzheimer's dementia (n= 36), vascular dementia (n= 9), frontotemporal dementia (n= 5), mixed dementia (combined Alzheimer's dementia and vascular dementia, n= 6), Lewy body dementia (n= 1) and psychiatric disorders (n= 9), whereas an additional nine patients were categorized as ‘other’. These included patients with varying diagnoses, such as cognitive impairment not further specified. Together, these patients formed the group with a clinical diagnosis and cognitive impairment.
A reference group was included consisting of 37 healthy participants without objective clinical or cognitive impairment. These participants were without a history of stroke, neurodegenerative or psychiatric disorders and with intact activities of daily living as indicated with a maximum score on the Barthel Index. Demographics of the two groups are shown in Table 1. The MMSE13 was given as a test of general cognitive functioning. Premorbid IQ was estimated with the Dutch version of the National Adult Reading Test.19 Educational attainment was rated with an ordinal rating scale ranging from 1 (less than primary school) to 7 (university degree).20 Non-parametric testing showed that the groups were comparable on age (U= 2478.0, P= 0.69) and education (U= 2187.5, P= 0.14). Similarly, no sex differences were noted between the groups (χ2(1) = 1.00, P= 0.32). Estimated IQ, however, was significantly lower in the cognitively impaired group (U= 852.5, P < 0.05). Although premorbid IQ is generally considered to be insensitive to the effects of age and mild to moderate cognitive decline,21 others report a strong association with dementia severity,22 which might account for the lower IQ estimate in our clinical group.
Table 1. Characteristics of the participants
Age, education, IQ and MMSE scores are expressed as mean (±standard deviation). Sex is defined as the number of males and females. Education was measured using an ordinal rating scale ranging between 1 (incomplete primary school) and 7 (university). MMSE, Mini-Mental State Examination.
This retrospective study was carried out in accordance with the Declaration of Helsinki and the guidelines for Good Clinical Practice established by the International Conference on Harmonisation (CPMP/ICH/135/95). Data were collected as part of the clinical routine and the confidentiality of participants' identities was maintained throughout the study process.
The Key Search Test, as an ecological test of executive function,5 measures planning ability and search strategy. Performance on this test was scored according to instructions in the test's manual and can range between 0 (least efficient search strategy) and 16 (most efficient search strategy). These raw scores can then be converted to a profile score (range 0–4).5
The following executive function tests, in addition to the Key Search Test, were chosen as these tests are commonly used neuropsychological tests at the memory clinic: the Trail Making Test part B (TMT-B), category fluency (animals and professions), the Brixton Spatial Anticipation Test,23 and the Rey Complex Figure Test.24,25 Performance on the TMT-B can be interpreted as a measure of flexibility; a score corrected for speed (TMT-B/TMT-A) was used for the analyses. Category fluency measures control and search strategy of word production; the total number of animals and professions generated within 1 min was considered. The Brixton Spatial Anticipation Test measures rule induction and set shifting; the number of errors was noted. Finally, the copy condition of the Rey Complex Figure Test measures planning and organizational capacity.26 Additionally, the Dutch version27 of the Rey Auditory Verbal Learning Task28 was included as an indicator of episodic memory performance. Both the total immediate recall and the delayed recall score were considered.
Spearman's rank correlations were calculated to examine compatibility between the Key Search Test and the other executive function tests. In order to detect the presence of executive function deficits, the following steps were undertaken. First, average performance and standard deviation of the control group was calculated for each separate task. Performance of the patients was compared with these average scores and considered to be impaired if their performance was 1.5 or more standard deviations below the performance of the control group.17 Executive function deficits were considered present if performance was impaired on one or more of the executive function tests. A receiver operating characteristic (ROC) curve was calculated to determine the discriminant validity of the Key Search Test (raw score) for executive function deficits, including all participants. This type of analysis produces an area under the curve (AUC), which indicates the discriminative power of the test. This value ranges between 0.5 (no discriminative power) to 1.0 (maximum discriminative power).
To examine divergent validity of the Key Search Test, we investigated whether the predictive value of this test was indeed better for executive functions than for memory performance. This was accomplished by examining the predictive value of the Key Search Test for memory impairment, which was based on an impaired total recall and/or an impaired delayed recall score of the Rey Auditory Verbal Learning Test. Again, for all participants, impaired performance was defined as performance being reduced by 1.5 standard deviations or more, based on the average performance of the control group.
To examine whether the Key Search Test has any additional discriminative power next to the predictive value of the MMSE in distinguishing between cognitive intact (i.e. the controls) and cognitive impaired (i.e. dementia and MCI patients) subjects, the AUC of the MMSE was compared with the AUC of the MMSE added to the Key Search Task profile score. For this latter analysis, we did not include the groups with ‘psychiatric diagnoses’ or ‘other diagnoses’.
In the total study sample, executive dysfunctioning was present in 50.8% of the participants. Correlation analysis showed small to medium correlations29 between the Key Search Task and other executive function measures (Table 2). ROC analysis of the Key Search Test as a predictor of executive function deficits showed an AUC of 0.677 (P < 0.001) (see Fig. 2). No optimal cut-off point could be determined (see Table 3) that fulfilled the criteria of a sensitivity of >80% and a specificity >60%.30 In order to examine the validity of the Key Search Test as a measure of executive ability, the predictive value of this test for memory dysfunction was examined. ROC analysis showed an AUC of 0.681 (P < 0.001), which is comparable to the AUC of the Key Search Test as a predictor of executive function impairment.
Table 2. Correlations between the Key Search and other executive function tests, estimated IQ and educational attainment
* P < 0.05; ** P < 0.01; *** P < 0.001. Spearman's correlations were calculated between the Key Search Test and other executive function tests, IQ and education. Performance on the BSAT reflects the number of errors. Trail Making Test B/Trail Making Test B (TMTB/TMTA) reflects the TMTB completion time divided by the TMTA completion time. BSAT, Brixton Spatial Anticipation Test; RCFT, Rey Complex Figure Test; TMT, Trail Making Test.
Table 3. Possible cut-off points of the Key Search Test for executive function deficits
A final aim was to examine whether the Key Search Test provides diagnostic value additional to the MMSE in differentiating between elderly people with and without cognitive impairment. ROC analysis of the MMSE showed an AUC of 0.808 (P < 0.001). When examining the added value of the Key Search Test to the MMSE in classifying cognitive impairment, an AUC of 0.823 (P < 0.001) was observed. This AUC did not differ significantly from the AUC of the MMSE alone (P= 0.59; see Fig. 3).
The present study aimed to examine the diagnostic accuracy of the Key Search Test as an indicator of executive function deficits and added to the MMSE as an indicator of overall cognitive decline. Although the Key Search Test acceptably discriminated between those participants with and without executive function impairment, we could not identify an optimum cut-off score. That is, sensitivity and specificity values were generally low. As such, a tentative conclusion is that the Key Search is not an accurate indicator of executive function impairment. In addition, the Key Search Test differentiated comparably between intact and impaired memory function, indicating a lack of divergent validity. This is supported by the small to medium correlations between this test and our other executive function tests. This finding is in agreement with previous studies, in which small to medium correlations between the Key Search Task and other executive function tests were also reported.7,31
Finally, we have assessed the additional value of the Key Search Test next to the MMSE in classifying cognitive impairment, because the MMSE itself does not contain any measure of executive function. The importance of including executive function measures in classifying cognitive impairment is corroborated by previous studies showing that executive function tasks might be very useful in predicting conversion to dementia1 as well as in discriminating between demented and non-demented people.14 The results, however, showed no additive value of the Key Search Test to the discriminant power of the MMSE. Apparently, the Key Search Test, at least in the present study group, does not involve cognitive functions other than those assessed with the MMSE. It is important to note, though, that in other populations it might indeed be the case that the Key Search Test is useful in addition to the MMSE when distinguishing between intact and impaired cognitive functioning. Future studies are in need to appropriately address this possibility.
The low discriminative value of the Key Search Test might also be due to a low construct validity. That is, correlations between this test and other executive function measures in the present study were modest. Furthermore, as its origins can be found in the Stanford-Binet Intelligence Scale, it might be that general intelligence, instead of executive function, is the main factor underlying test performance. A previous study observed significant associations between the Key Search Test and intellectual functions,7 and small to moderate correlations were also noted of the Key Search tests with both premorbid IQ and the level of educational attainment (see Table 2) in the present study. This supports the notion that intelligence is an important determinant of task performance. It should be noted, though, that the IQ estimate and educational background were both moderately correlated with most executive function measures in the present study (data not shown), showing a common relationship between intelligence and executive function, instead of a specific association with the Key Search Test.
Additionally, an important issue is whether a single test will ever be sufficiently capable to fully capture executive function performance. By definition, the term executive functions encompasses a wide variety of cognitive capacities that enable us to plan, monitor and execute a sequence of goal-directed complex actions.24 This multifaceted nature implies that tests measuring executive functions tap various capabilities and therefore heterogeneity among these tests might be substantial. The low to moderate correlations among the executive function tests in the present study support this postulation. As such, the present findings might not be task-specific, but might apply to executive function tasks in general. An important issue for future studies would be to examine the predictive value of single executive function tests for overall impaired executive ability.
To conclude, the Key Search Test could not accurately distinguish between intact and impaired executive function in the present study population. This observation, together with small to moderate correlations between this test and other executive tasks, suggests that the Key Search test is not an appropriate screening measure of executive functions in cognitive impaired individuals referred to memory clinics. This lack of diagnostic accuracy needs to be confirmed in different clinical populations.
We thank Nelleke van Schuylenborgh-van Es and Liesbeth Joosten-Weyn Banningh for assisting in examining the patients.