SEARCH

SEARCH BY CITATION

Keywords:

  • cognition;
  • psychiatric emergency services;
  • mass screening;
  • sensitivity and specificity;
  • differential diagnosis

Abstract

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. References

Objectives:  In certain clinical contexts, the sensitivity of the Mini-Mental State Examination (MMSE) is limited. The authors developed a new cognitive screening instrument, the Brief Cognitive Screen (BCS), with the aim of improving diagnostic accuracy for cognitive dysfunction in the psychiatric emergency department (ED) in a quick and convenient format.

Methods:  The BCS, consisting of the Oral Trail Making Test (OTMT), animal fluency, the Clock Drawing Test (CDT), and the MMSE, was administered to 32 patients presenting with emergent psychiatric conditions. Comprehensive neuropsychological evaluation served as the criterion standard for determining cognitive dysfunction. Diagnostic accuracy of the MMSE was determined using the traditional clinical cutoff and receiver operating characteristic (ROC) curve analyses. Diagnostic accuracy of individual BCS components and BCS Summary Scores was determined by ROC analyses.

Results:  At the traditional clinical cutoff, MMSE sensitivity (46.4%) and total diagnostic accuracy (53.1%) were inadequate. Under ROC analyses, the diagnostic accuracy of the full BCS Summary Score (area under the curve [AUC] = 0.857) was comparable to the MMSE (AUC = 0.828). However, a reduced BCS Summary Score consisting of OTMT Part B (OTMT–B), animal fluency, and the CDT yielded classification accuracy (AUC = 0.946) that was superior to the MMSE.

Conclusions:  Preliminary findings suggest the BCS is an effective, convenient alternative cognitive screening instrument for use in emergent psychiatric populations.

ACADEMIC EMERGENCY MEDICINE 2010; 17:307–315 © 2010 by the Society for Academic Emergency Medicine

The prevalence of dementia syndromes, particularly Alzheimer’s disease, is both substantial1 and increasing with demographic aging.2 Cognitive dysfunction is also a central feature of many types of medical and major psychiatric disorders3–6 and is associated with impaired adaptive function,7 diminished capacity to make decisions regarding medical treatment,8 and a higher risk of adverse treatment outcomes.9 Yet, evidence suggests that in a variety of clinical settings, the initial identification of cognitive disorder may be surprisingly ineffective.10–13 The Mini-Mental State Examination (MMSE)14 is the quintessential cognitive screen, but significant limitations have been identified, including low sensitivity15,16 and weak overall diagnostic accuracy.17–19 Various strategies to improve the MMSE or circumvent its limitations have met with variable success.20–26

The vast preponderance of the data regarding cognitive screening instruments has been gathered in outpatient clinics and samples of community-dwelling elders. In contrast, the utility of cognitive screening measures in the emergency department (ED) has been examined on a rather limited scale. Disorders of cognition are prevalent among patients presenting to the ED but, as in other populations, they are often poorly identified.27–30

Although psychometric limitations of available cognitive screening instruments are a relevant issue, perhaps a more important source of the difficulty in identifying cognitive disorders in the ED has been attributed to the perception that administering most such instruments is impractical.10,13 Emergency physicians (EPs) tend to regard the typical formal mental status examination as being excessively lengthy.31 Nonetheless, the majority recognize the need for a measure, preferably requiring 5 minutes or less to complete.32 However, attempts to balance efficacy and practicality of assessing cognition in the ED have been inconsistent.33–36 Moreover, the ED is a critical point of contact with the health care system and source of referrals for older and underserved populations.37

Collectively, then, the data indicate a clear need for an appropriate screening instrument to improve identification of cognitive disorders in many clinical contexts, particularly the ED. In addition to brevity, the ideal cognitive screen should be easy to score, relatively independent of confounding variables, and well tolerated by patients and should demonstrate acceptable psychometric properties.38 In our opinion, the sensitivity of any screening instrument is of preeminent concern. Although high levels of sensitivity and specificity are both clearly desirable within a single instrument, we argue that potential failure to render appropriate treatment due to false-negative error is considerably more serious than are the implications of false-positive error, which affect resource allocation and may be rectified by more detailed, specific follow-up testing.

Based on those guiding principles, we developed a new instrument, designated the Brief Cognitive Screen (BCS), with the aim of improving diagnostic accuracy in a format that can be deployed conveniently in the ED setting. Administration of the BCS is very brief and requires a minimum of materials, and each of its four components is scored quickly, easily, and independently from the others. We predicted that the BCS would demonstrate superior sensitivity and total diagnostic accuracy compared to the MMSE.

Methods

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. References

Study Design

This was a comparative study of the BCS and the MMSE. Because the BCS was administered as a routine aspect of clinical care, the requirement for written informed consent was waived by the institutional review boards of Bellevue Hospital Center and the New York University School of Medicine.

Study Setting and Population

The study was conducted in the Comprehensive Psychiatric Emergency Program (CPEP) at Bellevue Hospital Center in New York City. CPEP has an annual volume of approximately 10,000 patients, serving all of Manhattan and surrounding boroughs. The CPEP patient population has a median age of 38 years and is demographically diverse. Individuals from lower socioeconomic strata with severe psychiatric illness and substance-related disorders are over-represented.

Study Protocol

All patient data included in the study were retrospective and deidentified. Patients whose primary language was not English were excluded if their English language proficiency was insufficient to allow for adequate comprehension of instructions to engage in the BCS.

Standard clinical practice in CPEP is for the BCS to be administered to a subset of patients whose history suggests an elevated risk of cognitive dysfunction. Elements of the history that trigger BCS screening include spontaneous complaint of memory problems, age greater than 59 years, late-onset psychiatric disorder, or history of blunt head trauma. Patients presenting with acute alcohol or substance intoxication are not given the BCS because of the likelihood of confounding. Patients with a secondary diagnosis of alcohol or substance abuse who presented to CPEP without evidence of intoxication or withdrawal were eligible for inclusion. No patient who underwent BCS screening had any known documented evidence of cognitive dysfunction.

Eighty patients presenting to CPEP over a 3-year period underwent BCS screening. The BCS was conducted either by an experienced psychologist or by a psychology intern or psychiatry resident under the direct supervision of the psychologist. Of those 80 patients, 32 underwent criterion comprehensive neuropsychological assessment as part of their psychiatric treatment at Bellevue. Primary psychiatric diagnoses of the 32 patients referred for neuropsychological assessment included psychotic disorders (schizophrenia, schizoaffective disorder, psychotic disorder not otherwise specified; n = 18), mood disorders (depression, bipolar disorder; n = 13), and anxiety disorder (n = 1). In addition, five had a prior history of alcohol or substance abuse, and 15 had a prior history of blunt head trauma.

Forty-eight patients who underwent BCS screening did not complete neuropsychological evaluation, either at the inpatient attending physician’s discretion or because the patient was discharged prior to obtaining neuropsychological evaluation. Some patients who screened negative on the BCS were nonetheless referred for neuropsychological evaluation. The decision to refer in such cases was determined on clinical grounds and the uncertainty in the diagnostic accuracy of the BCS because it was still under development. In those cases, demographic characteristics (educational attainment, occupation) suggested a high level of premorbid functioning, which raised concern for the possibility of a false-negative BCS.

Measures

MMSE and BCS.  The MMSE was administered and scored according to procedures described previously.14 The amount of time required to administer the MMSE is typically 6 to 10 minutes.

The BCS has four components: the two-part Oral Trail Making Test (OTMT),39 animal fluency,40 and the Clock Drawing Test (CDT).41 Individually, all of the BCS components are previously published, well-established tests of cognitive function that are used frequently in neuropsychological assessment. Each component is scored quickly, easily, and independently from the others. In its totality, administration of the BCS is brief, typically about 4 minutes, and requires a minimum of materials, consisting of blank paper, pen, and timer.

The OTMT consists of two parts. Part A (OTMT–A) requires the patient to count rapidly from 1 to 25; the patient is timed to completion. Part B (OTMT–B) requires repeated shifting between two cognitive sets; the patient is instructed, “I’d like you to count again, but now I want you to switch back and forth between numbers and letters, for example, 1–A–2–B–3–C, and so on. Keep going until I ask you to stop.” Test performance proceeds through number 13 and is timed to completion. If the patient is unable to acquire set or proceed beyond 3–C, the test is terminated and a score of 300 seconds is assigned to the performance. For cases in which the patient progressed beyond the clinician-provided example (1–A, 2–B, 3–C) but was subsequently unable to complete the test in its entirety (ending 12–L–13), his or her performance was prorated to yield an estimated time to completion. The OTMT–B consists of 25 individual steps (e.g., 1–A = 2 steps); estimated time to completion was calculated by multiplying the interval time (in seconds) at discontinuation by 25 and dividing that result by the number of steps actually completed (e.g., . . . 5–E–6 = 11 steps; . . . 8–H = 16 steps).

Instructions for animal fluency were, “Name as many animals as quickly as possible. They can be from the farm, ocean, jungle, even house pets.” The total animal fluency score represented the number of animals generated within a 1-minute interval. Repetitions were not included in the total score, but superordinate categories and category exemplars were considered valid responses.

The CDT requires the patient to generate a graphomotor representation of a clock face with a specific designated time. In the current administration, the patient was instructed, “Draw a clock and put all the numbers where they belong” and subsequently “Set the clock for 10 minutes past 11.” The CDT was scored using a 5-point system; 1 point each was given for 1) an adequate contour (size, circularity); 2) presence of all 12 numbers; 3) correct positioning of all numbers within appropriate contour quadrants; and correct placement of both the 4) hour and the 5) minute hands.

A positive screen on the BCS was defined as a score falling within the deficient range or lower on at least one component of the BCS. Deficiency on OTMT–A, OTMT–B, and animal fluency was determined by converting the raw score to an appropriate Z-score using published norms.39,42 A Z-score of −1.0 or less was considered deficient. For the CDT, a raw score of <5 was considered deficient. Those definitions of deficiency conform to levels used in most clinical contexts and so exist independently of data analyses conducted in this study.

Criterion Neuropsychological Evaluation.  Following administration of the BCS, patients underwent comprehensive neuropsychological evaluation, which served as the criterion for determining group membership. Participants were given the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS)43 by a psychology trainee under the supervision of a neuropsychologist. Although the trainee was not blinded to the outcome of the BCS screen, criterion evaluation was conducted independently from staff administering the BCS. The mean interval between administration of the BCS and criterion neuropsychological evaluation was 20.3 days (SD ± 33.9 days). Definitive evidence of cognitive deficit was defined as a scaled score of ≤85 (Z-score ≤−1.0) on the RBANS total score or on any single RBANS subscale score.

Data Analysis

Analyses were conducted using SPSS release 10.0.1 (SPSS Inc., Chicago, IL, 1999). MMSE diagnostic accuracy was determined at the traditional 23/24 clinical cutoff score by examining its concordance with criterion comprehensive neuropsychological assessment. For comparison, each patient’s MMSE raw score was converted to a Z-score using published age- and education-corrected norms.20 Concordance between the MMSE and the criterion was determined for a set of Z-score cutoffs (−1.0, −1.5, and −2.0). Under both approaches, standard diagnostic decision tables were generated. Confidence intervals (CIs) were calculated using the adjusted Wald method for small samples.44

The diagnostic accuracy of the BCS was determined by receiver operating characteristics (ROC) curves, which were generated from raw scores for each BCS component as well as a BCS Summary Score. The Summary Score was calculated by transforming stratified Z-score ranges for OTMT–A, OTMT–B, and animal fluency into discrete levels of impairment (LOI). For those three BCS components, Z-scores of more than −1.0 (within normal limits) were assigned an LOI of 0; similarly, a Z-score range of −1.00 to −1.49 was assigned an LOI of 1, Z-scores from −1.50 to −1.99 were assigned an LOI of 2, Z-scores from of −2.00 to −2.49 were assigned an LOI of 3, Z-scores from −2.50 to −2.99 were assigned an LOI of 4, and Z-scores of −3.00 or less were assigned an LOI of 5. For the CDT, a raw score of 5 was assigned an LOI of 0, a score of 4 an LOI of 2, a score of 3 an LOI of 3, a score of 2 an LOI of 4, and a score of 0 or 1 was assigned an LOI of 5. The LOI for each BCS component was then summated to yield the BCS Summary Score. Thus, higher Summary Scores reflect increasing levels of impairment. Finally, ancillary analyses to examine the influence of confounding variables potentially mediating MMSE and BCS performance were conducted with Spearman rho (ρ) correlation coefficients.

Results

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. References

The study sample consisted of 11 women (34.4%) and 21 men. Patients were diverse in terms of age, education, and ethnicity. The mean age at BCS administration was 49.8 years (SD ± 13.9 years), ranging from 21 to 79 years. Seven patients (22%) held a high school diploma. Thirteen (41%) did not complete high school, whereas 12 (37%) had completed at least 1 year of college; five (16%) had a baccalaureate or postbaccalaureate degree. Eleven patients (34%) were Black or African American, nine (28%) were white, eight (25%) were Hispanic, and four (13%) were of other backgrounds.

The subgroup of patients given the BCS but not completing criterion neuropsychological assessment did not differ reliably from the BCS validation sample with respect to age (t = −0.662, p = 0.510), educational attainment [χ2(df = 6) = 4.09, p = 0.665], sex [χ2(df = 1) = 1.82, p = 0.203], ethnicity [χ2(df = 5) = 2.92, p = 0.712], primary language [χ2(df = 4) = 4.02, p = 0.403], or ED psychiatric diagnostic category [χ2(df = 6) = 2.4692, p = 0.873]. Moreover, the subgroup not completing criterion neuropsychological assessment did not differ in terms of MMSE total score (t = −0.404, p = 0.687) or performance on any BCS component, including animal fluency (t = 0.454, p = 0.651), the OTMT–A (t = −0.759, p = 0.450), the OTMT–B (t = −1.12, p = 0.263), or the CDT (t = −1.16, p = 0.251).

Diagnostic Accuracy of the Traditional Clinical MMSE Cutoff

At the traditional cutoff score of 23 of 24, MMSE sensitivity was 46.4% (95% CI = 29.5% to 64.2%), and total diagnostic accuracy was 53.1% (95% CI = 36.4% to 69.1%); specificity was 100% (lower limit of 95% CI = 54.3%). In contrast, using a normative correction approach, sensitivity (75.0%; 95% CI = 56.4% to 87.6%) and total diagnostic accuracy (75.0%; 95% CI = 57.7% to 87.0%) of the MMSE were optimal at the Z = −1.0 cutoff; specificity was also 75% (95% CI = 28.9% to 96.6%). At the −1.5 Z-score cutoff, sensitivity dropped to 64.3% (95% CI = 45.8% to 79.4%) and total diagnostic accuracy was 65.6% (95% CI = 48.2% to 79.7%); specificity remained stable at 75%. At the Z = −2.0 cutoff, all diagnostic accuracy measures were identical to the traditional clinical cutoff.

ROC

The proportion of area under the curve (AUC) for the MMSE was 0.828, reliably better than chance (Table 1). For the BCS Summary Score, AUC was 0.857, also better than chance. Considering individual BCS components, the OTMT–B had the highest AUC (0.911) of all individual tests; AUC for animal fluency (0.875) was comparable to that of the MMSE. In contrast, AUCs for the OTMT–A (0.652) and the CDT (0.665) failed to reach statistical significance and were lower than the AUC for the MMSE (Figure 1). Based on the results for individual BCS components, a reduced BCS Summary Score omitting OTMT–A was evaluated. In that case, ROC analyses yielded excellent classification accuracy; AUC was 0.946, which was superior to the MMSE and the full BCS Summary Score (Figure 2). Sensitivity and specificity values at specific cutoff scores for the MMSE are shown in Table 2, for BCS components in Tables 3 through 6, and for the reduced BCS Summary Score in Table 7.

Table 1.    Comparative Measures from ROC analyses
MeasureAUCSEp-value95% CI
  1. AUC = area under the ROC curve; BCS = Brief Cognitive Screen; CDT = Clock Drawing Test; MMSE = Mini-Mental State Examination; OTMT–A = Oral Trail Making Test Part A; OTMT–B = Oral Trail Making Test Part B; ROC = receiver operating characteristic curve.

  2. *Statistically significant at the p < 0.05 level.

MMSE0.828*0.1010.0360.629–1.0
BCS component
 Animal fluency0.875*0.0600.0160.757–0.993
 CDT 0.6650.1470.2920.377–0.953
 OTMT–A0.6520.1590.3330.340–0.964
 OTMT–B0.911*0.0620.0090.790–1.0
BCS Summary Score
 Full0.857*0.0760.0230.709–1.0
 Reduced0.946*0.0410.0040.866–1.0
image

Figure 1.  Comparative ROC curves for MMSE and individual BCS components. BCS = Brief Cognitive Screen; CDT = Clock Drawing Test; CI = confidence interval; MMSE = Mini–Mental State Examination; OTMT–A = Oral Trail Making Test Part A; OTMT–B = Oral Trail Making Test Part B; ROC = receiver operating characteristic curve.

Download figure to PowerPoint

image

Figure 2.  Comparative ROC curves for MMSE and BCS Reduced Summary Score. BCS = Brief Cognitive Screen; MMSE = Mini-Mental State Examination; ROC = receiver operating characteristic curve.

Download figure to PowerPoint

Table 2.    Diagnostic Accuracy Measures at Specified Cutoff Values for MMSE Raw Score
CutoffSensitivity (%)Specificity (%)
  1. MMSE = Mini-Mental State Examination.

<301000
<2910025
<2886.250
<2782.875
<2672.475
<2558.675
<2448.3100
<2344.8100
<2227.6100
<2120.7100
Table 3.    Diagnostic Accuracy Measures at Specified Cutoff Values for the OTMT–B Time to Completion (Seconds)
CutoffSensitivity (%)Specificity (%)
  1. OTMT–B = Oral Trail Making Test Part B.

>171000
>1910025
>2296.425
>2496.450
>2892.950
>3392.975
>3889.375
>4285.775
>4582.175
>4875.075
>5575.0100
>6864.3100
>8257.1100
>8953.6100
>10550.0100
>12346.4100
>21342.9100
>3010100
Table 4.    Diagnostic Accuracy Measures at Specified Cutoff Values for Animal Fluency (Number Generated in 1 Minute)
CutoffSensitivity (%)Specificity (%)
<271000
<2599.60
<2493.10
<2393.125
<2189.725
<1986.275
<1782.875
<1679.3100
<1572.4100
<1465.5100
<1358.6100
<1251.7100
<1141.4100
<1024.1100
<810.3100
Table 5.    Diagnostic Accuracy Measures at Specified Cutoff Values for CDT (Total Score)
CutoffSensitivity (%)Specificity (%)
  1. CDT = Clock Drawing Test.

<582.550
<452.550
<342.575
<227.5100
<17.5100
Table 6.    Diagnostic Accuracy Measures at Specified Cutoff Values for OTMT–A Time to Completion (Seconds)
CutoffSensitivity (%)Specificity (%)
  1. OTMT–A = Oral Trail Making Test Part A.

>41000
>610025
>792.925
>878.650
>957.150
>1048.3100
>1146.4100
>1242.9100
>1328.6100
>1425.0100
>1521.4100
>197.1100
Table 7.    Diagnostic Accuracy Measures at Specified Cutoff Values for the Reduced BCS Summary Score
Cut-offSensitivity (%)Specificity (%)
  1. BCS = Brief Cognitive Screen.

>110025
>392.925
>492.950
>592.9100
>685.7100
>782.1100
>867.9100
>950.0100
>1032.1100
>1221.4100
>1314.3100
>1410.7100

Influence of Confounding Variables

Total score from the MMSE showed significant correlation with education (ρ = 0.386, p = 0.029) but not with age (ρ = −0.064, p = 0.726). No significant correlations were found between age or education and any BCS component.

Given the mean interval of about 20 days between BCS and criterion neuropsychological assessment, a median split was performed to examine the potential for systematic variability in cognition due to treatment differences between subgroups with short versus long intervals. The median interassessment interval was 9 days; for the short-interval subgroup, specificity values were lower than for the long-interval subgroup at comparable sensitivity levels, indicating that the rate of false-positive errors was somewhat higher in the short-interval rather than the long-interval subgroup.

Discussion

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. References

Preliminary data indicate that among a small convenience sample of patients presenting with emergent psychiatric symptoms, the BCS offers excellent sensitivity and specificity in a format that requires only 4 minutes to administer. Those properties suggest that the BCS is an adequate alternative screening method for cognitive dysfunction.

The BCS redresses the problem of low sensitivity with the MMSE. In particular, our data show that the traditional clinical cutoff of 23/24 on the MMSE yields very low sensitivity among emergent psychiatric patients. Moreover, MMSE scores are substantially influenced by age, or as we show, by education, which may contribute to false positive errors and reduced specificity. Normative correction of MMSE scores improves its diagnostic accuracy, but introduces further demands on the clinician with respect to time and complexity of interpretation.

Specific components of the BCS were chosen rationally. First, each component met a criterion of practicality, consisting of brevity and simplicity of administration and interpretation. For an instrument to be used consistently, clinicians must feel that it represents a realistic solution to the demands of the clinical setting. However, many clinicians, including EPs,31 perceive the typical formal mental status examination as being excessively lengthy or inconvenient to administer and are therefore reluctant to screen routinely for cognitive disorders.10,13 Nonetheless, the majority of EPs report a need for, and would welcome, a measure of mental status requiring 5 minutes or less to complete.32 Second, the BCS as a whole attempts to reduce demands on the visual system. Low vision is a prevalent concern in elders45 and often limits the cognitive screening examination, including the MMSE.18 Finally, each component was intended to measure cognitive domains that may not be sampled adequately by the MMSE, with the aim of increasing sensitivity. The CDT and animal fluency were selected based on favorable prior evidence as brief measures of specific cognitive domains or as components of preexisting screening instruments. The CDT appears to have comparable sensitivity to the MMSE46 and similar measures,47 and predicts cognitive progression.21,38 Another advantage of the CDT is its relative insensitivity to confounding variables such as effects of primary languages other than English and cultural variation.48 However, it may still have limited sensitivity in identifying patients meeting criteria for mild cognitive impairment.48,49 Animal fluency was chosen based on evidence of its sensitivity for detecting Alzheimer’s disease.50,51 The OTMT,39 which has not been incorporated as a component of any previous cognitive screening instrument, was chosen to allow for quick but robust sampling of processing speed and the cognitive flexibility dimension of executive functions, which may not be measured adequately by the MMSE.

In contrast to the wide use of screening instruments in community-dwelling and clinic-based elders, data examining the efficacy of cognitive screening methods in the psychiatric ED are quite limited. In that setting, the CDT has been promulgated as a promising stand-alone cognitive screening measure.27 However, that study, consistent with our findings, obtained an ROC for the CDT that was comparable to the MMSE; the CDT is therefore probably insufficient as a stand-alone screen. Among other measures used in the psychiatric ED, the Wechsler Memory Scale,52 the Visual Reproduction (VR) subtest, and the Trail Making Test53 Part A (TMT–A) were reliable predictors of the likelihood of admission from psychiatric ED to an inpatient unit and clinical diagnosis at discharge, respectively.54 Among ED patients with comorbid schizophrenia or cocaine abuse, the California Verbal Learning Test (CVLT)55 revealed worse memory dysfunction in dual-diagnosis patients compared to single-diagnosis groups, which did not differ from each other.56 However, the intent of the latter study was to determine the presence of group differences rather than to screen for cognitive dysfunction. No diagnostic accuracy data were reported in either study. Moreover, the VR, TMT–A, and CVLT are each time-intensive and require specific materials, instruction sets, and reference to detailed norms for interpretation, all of which presume specialized training. Thus, those tests are clearly impractical to apply systematically for use in most psychiatric ED settings.

Prior ED-based studies of cognitive screening have been limited by the use of independent screening instruments as the criterion measure for cognitive dysfunction.57 That practice risks underestimating the prevalence of cognitive dysfunction either through low screening sensitivity (false-negative screen and criterion) or low specificity (true-positive screen, false-negative criterion). Other studies have employed DSM diagnosis as the criterion measure.58 However, that practice is also problematic because it assumes a specious dichotomy; cognitive dysfunction is a core or prominent feature of many major psychiatric disorders, including major depressive disorders4–6 and schizophrenia,3 and reflects real underlying cerebral dysfunction. The result in that case may be an overestimation of false-positive errors and spuriously low specificity. In our opinion, comprehensive neuropsychological evaluation represents the sole criterion standard for determining the presence of subtle cognitive dysfunction in validation studies of cognitive screening instruments.

Cognitive screening in the psychiatric emergency setting may be particularly beneficial in the management of patients with psychiatric disorders.27 There is ample evidence that cognitive dysfunction in general, and executive dysfunction in particular, are critical factors contributing to problems with decision-making capacity,59–61 especially in the early to middle stages of cognitive progression.62 Cognitive impairment among elder ED patients is associated with higher health care costs compared to elders with intact cognition.37 Early, accurate identification of cognitive dysfunction among patients with major psychiatric disorders may ultimately enhance treatment outcomes of those patients.63 The BCS may provide a means to address and help mitigate problems associated with a lack of appropriate screening.

Limitations

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. References

The most pressing and obvious limitations of this study involve sampling issues. First, the sample size was limited to 32 participants. Low numbers may result in unstable psychometric properties, potentially reducing the predictive validity of the BCS. Second, substantial sample selection bias was present. Patients given the BCS, rather than consisting of an unselected sample, represented a convenience sample with an elevated risk of cognitive dysfunction. Accordingly, the sample had a low proportion of cognitively normal patients, which may have limited the reliability of the BCS specificity estimate. Although most patients included in the sample had a history of chronic psychiatric disorders, specificity may also have been limited by the presence of some patients with acute disorders or acute exacerbation of chronic disorders. Moreover, selection bias inherent to the CPEP patient population is a significant limitation. Conclusions regarding the generalizability of findings to general psychiatric or general ED populations must be constrained accordingly.

The lag between BCS screening and criterion neuropsychological evaluation may be viewed as problematic because it introduces the potential for confounding specific to treatment and from the nonspecific effects of the passage of time. However, we argue that a mean interval of 20 days generally reflects clinical reality and is consistent with such intervals in the relevant literature. Because most participants had chronic conditions, and those with acute intoxication were excluded, the likelihood of confounding was reduced. Confounding effects of treatment would manifest as a false-positive BCS. However, ROC analyses show that the number of false-positive errors was low overall. Moreover, a median split examining shorter versus longer treatment intervals showed that a false-positive BCS was somewhat more likely to occur in the shorter-interval subgroup. If treatment effects were a significant confound, a larger number of false-positive errors would be expected in the longer-interval subgroup because of the greater amount of time available for symptom reduction.

Test–retest reliability was not obtained. Thus, the repeatability of cognitive deficits as screened by the BCS in this population was not determined. In particular, the stability of cognitive deficit in patients with a primary diagnosis of mood or psychotic disorder may have been affected by acute new-onset or exacerbated conditions. Moreover, given the lack of a normative sample, it is also likely that if applied to a sample with a lower base rate of cognitive dysfunction, the BCS is likely to demonstrate lower sensitivity. Nonetheless, our goal at this initial stage of BCS development was to provide a direct comparison of its psychometric properties with those of the MMSE within a representative clinical context: patients presenting to the ED for treatment of emergent psychiatric disorders. Acknowledging those limitations, the data suggest that the BCS is effective in identifying cognitive dysfunction within that population. Cross-validation of the diagnostic accuracy in a larger, unselected sample will be accomplished in future research.

Conclusions

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. References

Preliminary findings indicate that the Brief Cognitive Screen improves substantially upon the clinical utility of the Mini-Mental State Examination for detecting cognitive dysfunction in selected emergent psychiatric populations. Moreover, it allows for effective cognitive screening in a format that is brief, simple, and convenient to administer. Those test characteristics may facilitate broader application of cognitive screening in emergent psychiatric populations, which in turn may lead to improved treatment outcomes in those patients.

The authors thank Alla Landa for assistance with data collection.

References

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. References
  • 1
    Hendrie HC. Epidemiology of dementia and Alzheimer’s disease [abstract]. Am J Geriatr Psychiatry. 1998; 6(2 Suppl 1):S318.
  • 2
    Evans DA. Estimated prevalence of Alzheimer’s disease in the United States. Milbank Q. 1990; 68:26789.
  • 3
    Bowie CR, Harvey PD. Cognition in schizophrenia: impairments, determinants, and functional importance. Psychiatr Clin North Am. 2005; 28:61333.
  • 4
    Ganguli M, Du Y, Dodge HH, Ratcliff GG, Chang CC. Depressive symptoms and cognitive decline in late life. A prospective epidemiological study. Arch Gen Psychiatry. 2006; 63:15360.
  • 5
    Rapp MA, Dahlman K, Sano M, Grossman HT, Haroutunian V, Gorman JM. Neuropsychological differences between late-onset and recurrent geriatric major depression. Am J Psychiatry. 2005; 162:6918.
  • 6
    Weiland-Fiedler P, Erickson K, Waldeck T, et al. Evidence for continuing neuropsychological impairments in depression. J Affect Disord. 2004; 82:2538.
  • 7
    Harvey PD, Leff J, Trieman N, Anderson J, Davidson M. Cognitive impairment in geriatric chronic schizophrenic patients: a cross-national study in New York and London. Int J Geriatr Psychiatry. 1997; 12:10017.
  • 8
    Katz M, Abbey S, Rydall A, Lowy F. Psychiatric consultation for competency to refuse medical treatment. A retrospective study of patient characteristics and outcome. Psychosomatics. 1995; 36:3341.
  • 9
    Breier A, Schreiber JL, Dyer J, Pickar D. National Institute of Mental Health longitudinal study of schizophrenia: prognosis and predictors of outcome. Arch Gen Psychiatry. 1991; 48:23946.
  • 10
    Brodaty H, Clarke J, Ganguli M, et al. Screening for cognitive impairment in general practice: toward a consensus. Alzheimer Dis Assoc Disord. 1998; 12:113.
  • 11
    Garcia CA, Tweedy JR, Blass JP. Underdiagnosis of cognitive impairment in a rehabilitation setting. J Am Geriatr Soc. 1984; 32:33942.
  • 12
    Knights EB, Folstein MF. Unsuspected emotional and cognitive disturbance in medical patients. Ann Int Med. 1977; 87:7234.
  • 13
    Rubin SM, Glasser ML, Werckle MA. The examination of physician’s awareness of dementing disorders. J Am Geriatr Soc. 1987; 35:10518.
  • 14
    Folstein MF, Folstein SE, McHugh PR. ‘Mini-mental state’. A practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res. 1975; 12:18998.
  • 15
    Tierney MC, Szalai JP, Dunn E, Geslani D, McDowell I. Prediction of probable Alzheimer disease in patients with symptoms suggestive of memory impairment. Arch Fam Med. 2000; 9:52732.
  • 16
    Wind AW, Schellevis FG, Van Staveren G, Scholten RJ, Jonker C, Van Eijk JT. Limitations of the Mini-Mental State Examination in diagnosing dementia in general practice. Int J Geriatr Psychiatry. 1997; 12:1018.
  • 17
    Anthony JC, LeResche L, Niaz U, Von Korff MR, Folstein MF. Limits of the ‘Mini-Mental State’ as a screening test for dementia and delirium among hospital patients. Psychol Med. 1982; 12:397408.
  • 18
    Raiha I, Isoaho R, Ojanlatva A, Viramo P, Sulkava R, Kivela SL. Poor performance on the mini-mental state examination due to causes other than dementia. Scan J Primary Health Care. 2001; 19:348.
  • 19
    Tombaugh TN, McIntyre NJ. The mini-mental state examination: a comprehensive review. J Am Geriatr Soc. 1992; 40:92235.
  • 20
    Crum RM, Anthony JC, Bassett SS, Folstein MF. Population-based norms for the Mini-Mental State Examination by age and educational level. JAMA. 1993; 269:238691.
  • 21
    Ferrucci L, Cecchi F, Guralnik JM, et al. Does the Clock Drawing Test predict cognitive decline in older persons independent of the Mini-Mental State Examination? J Am Geriatr Soc. 1996; 44:132631.
  • 22
    Lawrence J, Davidoff D, Katt-Lloyd D, Auerbach M, Hennen J. A pilot program of improved methods for community-based screening for dementia. Am J Geriatr Psychiatry. 2001; 9:20511.
  • 23
    Lorentz WJ, Scanlan JM, Borson S. Brief screening tests for dementia. Can J Psychiatry. 2002; 47:72333.
  • 24
    Robert PH, Schuck S, DuBois B, et al., for the Investigators’ Group Screening for Alzheimer’s disease with the Short Cognitive Evaluation Battery. Dement Geriatr Cogn Disord. 2003; 15:928.
  • 25
    Swearer JM, Drachman DA, Li L, Kane KJ, Dessureau B, Tabloski P. Screening for dementia in “real world” settings: the Cognitive Assessment Screening Test: CAST. Clin Neuropsychologist. 2002; 16:12835.
  • 26
    Tariq SH, Tumosa N, Chibnall JT, Perry MH, Morley JT. Comparison of the St. Louis University Mental Status Examination and the Mini-Mental State Examination for detecting dementia and mild neurocognitive disorder – a pilot study. Am J Geriatr Psychiatry. 2006; 14:90010.
  • 27
    Copersino ML, Serper M, Allen MH. Rapid screening for cognitive impairment in the psychiatric emergency service: II. A flexible test strategy. Psychiatric Serv. 2003; 54:3146.
  • 28
    Elie M, Rousseau F, Cole M, Primeau F, McCusker J, Bellavance F. Prevalence and detection of delirium in elderly emergency department patients. Can Med Assoc J. 2000; 163:97781.
  • 29
    Litovitz GL, Hedberg M, Wise TN, White JD, Mann LS. Recognition of psychological and cognitive impairments in the emergency department. Am J Emerg Med. 1985; 3:4002.
  • 30
    Hustey FM, Meldon SW, Smith MD, Lex CK. The effect of mental status screening on the care of elderly emergency department patients. Ann Emerg Med. 2003; 41:67884.
  • 31
    Fauman MA, Fauman BJ. The differential diagnosis of organic based psychiatric disturbance in the emergency department. JACEP. 1977; 6:31523.
  • 32
    Zun L, Gold I. A survey of the form of mental status examination administered by emergency physicians. Ann Emerg Med. 1986; 15:91622.
  • 33
    Huff JS, Farace E, Brady WK, Kheir J, Shawver G. The Quick Confusion Scale in the ED: comparison with the Mini-Mental State Examination. Am J Emerg Med. 2001; 19:4614.
  • 34
    Lamarre CJ, Patten SB. Evaluation of the Modified Mini-Mental State Examination in a general psychiatric population. Can J Psychiatry. 1991; 36:50711.
  • 35
    Merigian KS, Hedges JR, Roberts JR, Childress RA, Niehaus MA, Franklin N. Use of abbreviated mental status examination in the initial assessment of overdose patients. Arch Emerg Med. 1988; 5:13945.
  • 36
    Wilber ST, Lofgren SD, Mager TG, Blanda M, Gerson LW. An evaluation of two screening tools for cognitive impairment in older emergency department patients. Acad Emerg Med. 2005; 12:6126.
  • 37
    McCusker J, Jacobs P, Dendukuri N, Latimer E, Tousignant P, Verdon J. Cost-effectiveness of a brief two-stage emergency department intervention for high-risk elders: results of a quasi-randomized controlled trial. Ann Emerg Med. 2003; 41:4556.
  • 38
    Shulman KI. The Clock Drawing Test: is it the ideal cognitive screening test? Int J Geriatr Psychiatry. 2000; 15:54861.
  • 39
    Ricker J, Axelrod B. Analysis of an oral paradigm for the Trail Making test. Assessment. 1994; 1:4751.
  • 40
    Spreen O, Strauss E. A Compendium of Neuropsychological Tests, 2nd ed. New York, NY: Oxford, 1998.
  • 41
    Goodglass H, Kaplan E. The assessment of aphasia and related disorders. Philadelphia, PA: Lea & Febiger, 1972.
  • 42
    Heaton RK, Miller SW, Taylor MJ, Grant I. Revised comprehensive norms for an expanded Halstead-Reitan Battery: demographically adjusted neuropsychological norms for African American and Caucasian adults. Professional Manual. Lutz, FL: Psychological Assessment Resources, 2004.
  • 43
    Randolph C. RBANS: Repeatable Battery for the Assessment of Neuropsychological Status. San Antonio, TX: The Psychological Corporation, 1998.
  • 44
    Sauro J, Lewis JR. Estimating completion rates from small samples using binomial confidence intervals: comparisons and recommendations. In: Proceedings of the Human Factors and Ergonomics Society 49th annual meeting, September 26–30, 2005; Orlando, FL, pp 21004.
  • 45
    Buch H, Vinding T, La Cour M, Appleyard M, Jensen GB, Nielsen NV. Prevalence and causes of visual impairment and blindness among 9980 Scandinavian adults: the Copenhagen city eye study. Ophthalmology. 2004; 111:5361.
  • 46
    Manos PJ. Ten-point clock test sensitivity for Alzheimer’s disease in patients with MMSE scores greater than 23. Int J Geriatr Psychiatry. 1999; 14:4548.
  • 47
    Suhr JA, Grace J. Brief cognitive screening of right hemisphere stroke: relation to functional outcome. Arch Phys Med Rehabil. 1999; 80:7736.
  • 48
    Powlishta KK, Von Dras DD, Stanford A, et al. The clock drawing test is a poor screen for very mild dementia. Neurology. 2002; 59:898903.
  • 49
    Peterson A, Lantz MS. Is it Alzheimer’s? Neuropsychological testing helps to clarify diagnostic puzzle. Geriatrics 2001; 56:5861.
  • 50
    Cercy SP, Rich JB. Phonemic and Semantic Verbal Fluency in the Amnestic Syndrome and Alzheimer’s Disease. Paper presented at the 24th annual meeting of the International Neuropsychological Society, Chicago, IL, February 1996.
  • 51
    Henry JD, Crawford JR, Phillips LH. Verbal fluency performance in dementia of the Alzheimer’s type: a meta-analysis. Neuropsychologia. 2004; 42:121222.
  • 52
    Wechsler D. A standardized memory scale for clinical use. J Psychol. 1945; 19:8795.
  • 53
    Reitan RM, Wolfson D. The Halstead-Reitan Neuropsychological Test Battery. Tucson, AZ: Neuropsychology Press, 1985.
  • 54
    Galynker II, Harvey PD. Neuropsychological screening in the psychiatric emergency room. Compr Psychiatry. 1992; 33:2915.
  • 55
    Delis DC, Kramer JH, Kaplan E, Ober BA. California Verbal Learning Test: Adult Version Manual. San Antonio, TX: The Psychological Corporation, 1987.
  • 56
    Serper M, Bergman A, Copersino ML, Chou JC, Richarme D, Cancro R. Learning and memory impairment in cocaine dependent and comorbid schizophrenic patients. Psychiatry Res. 2000; 93:2132.
  • 57
    Irons MJ, Farace E, Brady WJ, Huff JS. Mental status screening of emergency department patients: normative study of the Quick Confusion Scale. Acad Emerg Med. 2002; 9:98994.
  • 58
    Lamarre CJ, Patten SB. A clinical evaluation of the Neurobehavioral Cognitive Status Examination in a general psychiatric inpatient population. J Psychiatry Neurosci. 1994; 19:1038.
  • 59
    Allen RS, DeLaine SR, Chaplin WF, et al. Advance care planning in nursing homes: correlates of capacity and possession of advance directives. Gerontologist. 2003; 43:30917.
  • 60
    Dymek MP, Atchison P, Harrell L, Marson DC. Competency to consent to medical treatment in cognitively impaired patients with Parkinson’s disease. Neurology. 2001; 56:1724.
  • 61
    Marson DC, Cody HA, Ingram KK, Harrell LE. Neuropsychologic predictors of competency in Alzheimer’s disease using a rational reasons legal standard. Arch Neurol. 1995; 52:9559.
  • 62
    Marson DC. Loss of competency in Alzheimer’s disease: conceptual and psychometric approaches. Int J Law Psychiatry. 2001; 24:26783.
  • 63
    Emsley R, Chiliza B, Schoeman R. Predictors of long-term outcome in schizophrenia. Curr Opin Psychiatry. 2008; 21:1737.