SEARCH

SEARCH BY CITATION

Keywords:

  • geriatrics;
  • mental status;
  • diagnostic testing;
  • cognitive impairment

Abstract

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. Acknowledgments
  9. References

Background:  Cognitive impairment due to delirium or dementia is common in older emergency department (ED) patients. To prevent errors, emergency physicians (EPs) should use brief, sensitive tests to evaluate older patient’s mental status. Prior studies have shown that the Six-Item Screener (SIS) meets these criteria.

Objectives:  The goal was to verify the performance of the SIS in a large, multicenter sample of older ED patients.

Methods:  A prospective, cross-sectional study was conducted in three urban academic medical center EDs. English-speaking ED patients ≥65 years old were enrolled. Patients who received medications that could affect cognition, were too ill, were unable to cooperate, were previously enrolled, or refused to participate were excluded. Patients were administered either the SIS or the Mini-Mental State Examination (MMSE), followed by the other test 30 minutes later. An MMSE of 23 or less was the criterion standard for cognitive impairment; the SIS cutoff was 4 or less for cognitive impairment. Standard operator characteristics of diagnostic tests were calculated with 95% confidence intervals (CIs), and a receiver operating characteristic curve was plotted.

Results:  The authors enrolled 352 subjects; 111 were cognitively impaired by MMSE (32%, 95% CI = 27% to 37%). The SIS was 63% sensitive (95% CI = 53% to 72%) and 81% specific (95% CI = 75% to 85%). The area under the receiver operating characteristic curve was 0.77 (95% CI = 0.72 to 0.83).

Conclusions:  The sensitivity of the SIS was lower than in prior studies. The reasons for this lower sensitivity are unclear. Further study is needed to clarify the ideal brief mental status test for ED use.

ACADEMIC EMERGENCY MEDICINE 2008; 15:613–616 © 2008 by the Society for Academic Emergency Medicine

Older emergency department (ED) patients present unique challenges to emergency physicians (EPs). Approximately 23% to 40% of older ED patients are cognitively impaired, either due to delirium or dementia.1–6 However, multiple studies have shown that this impairment is frequently unrecognized by EPs. The prevalence of delirium is approximately 10% in older ED patients, and it is detected by EPs in only 17% to 35% of cases.1–5 Patients with delirium undiagnosed in the ED have a mortality rate nearly three times higher than those in whom delirium is appropriately diagnosed.1 A recent editorial emphasized the importance of missed delirium in older ED patients and argued that this represented an area of “medical errors, missed diagnoses, and quality-of-care concerns.”7

Although missed delirium is certainly the most serious potential consequence of inadequate mental status screening, identifying chronic cognitive impairment is also important. More than 80% of ED patients with cognitive impairment have no prior history of dementia.4,8,9 Failure to recognize chronic cognitive impairment may lead to a host of difficulties for the EP, from limiting the reliability of patients’ history, to reducing their ability to understand and comply with discharge instructions.10 Consequently, improving EPs’ recognition of cognitive impairment, whether acute or chronic, is essential.

Multiple screening tests for the recognition of cognitive impairment have been proposed.11 The Mini-Mental State Examination (MMSE) was developed in 1975 and has since become a widely used and accepted method for recognizing cognitive impairment.12 This test assesses many cognitive domains including orientation, registration, attention and calculation, recall, language, and the ability to follow simple commands.12 However, the MMSE takes a median of 6 minutes to administer to older ED patients (range 3.5 to 14 minutes) and is difficult for clinicians to administer without scoring sheets.6 These issues make it impractical for routine use in the fast-paced environment of the ED. Tools feasible for routine ED use must be brief, sensitive, and easily remembered tests that can be incorporated into the routine history and exam, without substantially increasing the time needed to evaluate the patient. One brief cognitive screen that meets these criteria is the Six-Item Screener (SIS). The SIS consists of three questions on temporal orientation (day, month, and year), as well as three-item recall.13 It is simple, easy to remember, and easy to score (the score is the simple summation of correct responses), allowing it to be incorporated into the routine clinical care of older ED patients. In a preliminary single-center study, the sensitivity and specificity of the SIS were 94% and 86%, respectively, and it took a median of 1 minute to administer to ED patients.6 Our goal was to evaluate the sensitivity of the SIS for the detection of cognitive impairment in older ED patients, using the MMSE as a criterion standard, in a larger, multicenter sample.

Methods

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. Acknowledgments
  9. References

Study Design

This was a prospective, cross-sectional study in the EDs of three academic medical centers. The study was approved by the institutional review boards at all sites, with a waiver of informed consent. Verbal consent was obtained from patients or proxies.

Study Setting and Population

Summa Health System’s Akron City Hospital (Site 1) is a community teaching hospital with 72,500 ED visits each year; approximately 25% are by patients 65 years old or older. Washington University, Barnes-Jewish Hospital (Site 2), is an academic teaching hospital with 72,325 total visits, 18% of which are by patients 65 years old or older. The Cleveland Clinic (Site 3) is an urban tertiary care center with approximately 60,000 ED visits per year, 20% of which are by patients 65 years and older. We enrolled a convenience sample of ED patients aged 65 years old or older who were able to communicate in English. We excluded patients who received medications that may have affected their mental status during the testing period (such as narcotics, antiemetics, or benzodiazepines), were critically ill, were unable to consent or cooperate with data acquisition, were previously enrolled, or refused to complete the questioning.

Study Protocol

Patients were enrolled based on the availability of research assistants at all three study sites. Enrollment occurred on weekdays; weekends; and day, evening, and night shifts. For eligible and consenting patients, the physician evaluating the patient (either a senior EM resident or an attending) rated the patient’s level of consciousness and performed the SIS. Trained research personnel administered the MMSE using a standardized data collection form. The order of administration of the MMSE and SIS differed between sites. At Sites 1 and 3, the SIS was administered first, and the MMSE was administered a minimum of 30 minutes later. At Site 2, the MMSE was administered first, and the SIS was administered a minimum of 30 minutes later. This delay was to provide a washout period for the questioning, especially 3-item recall.

During their evaluation, physicians rated the patient’s level of consciousness as alert and attentive (normal), inattentive, hyperalert, lethargic, stuporous, or comatose. The physician also performed and scored the SIS during their routine history and examination. Upon completion, the physician placed the study sheet in an envelope, sealed it, and recorded the time on the outside of the envelope. Trained research personnel performed the MMSE according to standard instructions.12 Both the physicians performing the SIS and the research personnel completing the MMSE were blinded to the results of the other test.

Data Analysis

Standard operator characteristics of diagnostic tests—sensitivity, specificity, positive and negative predictive values (PPVs, NPVs), and likelihood ratios (LRs) with 95% confidence intervals (CIs)—were calculated. We used an MMSE score of 23 or less as the criterion standard for cognitive impairment and an SIS score of 4 or less as the cutoff for impairment by this test, based on prior studies.6,13

The sensitivity of the SIS in our preliminary, single-site study was 94%.6 Verifying this sensitivity with a lower limit of the 95% CI of 90% would require 138 cognitively impaired patients.

Results

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. Acknowledgments
  9. References

Between January 12, 2006, and January 14, 2007, we approached 492 subjects; of those, 110 refused to participate. Of those for whom a reason for exclusion was recorded, 5 were too ill, 9 were unable to consent, 8 left before completion of both tests, and 6 were administered medications that could affect their mental status during the testing period. This left 352 enrolled subjects (152 at Site 1, 152 at Site 2, and 48 at Site 3).

The mean (± standard deviation [SD]) age of subjects was 77 (±8) years. A total of 223 (63%) were female. Most patients (338, 96%) had a normal level of consciousness. Cognitive impairment by MMSE was present in 32% of subjects (111/352, 95% CI = 27% to 37%).

Overall, the SIS was 63% sensitive and 81% specific; the NPV was 83% and the PPV was 60% (Table 1). The overall agreement between the two tests was 75%. There was some variation by site, with Site 1 having the highest sensitivity and Site 2 having the highest specificity (Table 1). The area under the receiver operator curve was 0.77 (95% CI = 0.72 to 0.83; Figure 1).

Table 1.   Operator Characteristics of the SIS at Each Site and Overall
 Sensitivity, % (95% CI)Specificity, % (95% CI)PPV, % (95% CI)NPV, % (95% CI)LR+ (95% CI)LR− (95% CI)
  1. CI = confidence interval; LR = likelihood ratio; NPV = negative predictive value; PPV = positive predictive value; SIS = Six Item Screener.

Site 168 (25/37) (50, 82)76 (87/115) (67, 83)47 (25/53) (33, 61)68 (87/99) (80, 94)2.8 (1.8, 4.1)0.4 (0.3, 0.7)
Site 260 (36/60) (47, 72)88 (81/92) (79, 94)77 (36/47) (62, 88)77 (81/105) (68, 85)5.0 (2.8, 9.1)0.5 (0.3, 0.6)
Site 364 (9/14) (35, 87)76 (26/34) (59, 89)53 (9/17) (28, 77)84 (26/31) (66, 95)2.7 (1.3, 5.6)0.5 (0.2, 1.0)
Overall63 (70/111) (53, 72)81 (194/241) (75, 85)60 (70/117) (50, 69)83 (194/235) (77, 87)3.2 (2.4, 4.3)0.5 (0.4, 0.6)
image

Figure 1.  Receiver operator curve (ROC) analysis of SIS, with MMSE as gold standard. MMSE = Mini-Mental State Examination; SIS = Six Item Screener.

Download figure to PowerPoint

Because of the unexpectedly low sensitivity, we performed an unplanned secondary analysis of the data. In this analysis, we calculated an SIS score by adding together its components from the MMSE (month, day, year, recall). This analysis demonstrated discordance in the classification of cognitive impairment between the measured SIS and the calculated SIS in 100 subjects (28%). The sensitivity of the calculated SIS was 85% (94/111, 95% CI = 77% to 91%) and the specificity was 71% (170/241, 95% CI = 64% to 76%).

Discussion

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. Acknowledgments
  9. References

In this multicenter study, the sensitivity of the SIS to detect cognitive impairment (defined as a MMSE < 23) was lower than in prior studies. The original study evaluated community-dwelling elders and an Alzheimer’s clinic population in a non-ED setting. In that study, the SIS was both sensitive and specific for the diagnosis of cognitive impairment.13 The performance of the SIS and the MMSE were comparable, suggesting that the SIS could be used as a sole screening test for cognitive impairment.13

In a smaller preliminary study in the ED at Site 1, using similar methodology, we found that the sensitivity of the SIS was 94% (95% CI = 73% to 100%) and specificity was 86% (95% CI = 74% to 94%).6 There are several possibilities for the differences found in our study. First, it is possible that there was a learning phenomenon for the questioning—that subjects scored higher on their second attempt at the questioning than on the first attempt. Such an effect has been previously described in dementia research.14,15 This would explain the lower sensitivity and higher specificity at Site 2, where the MMSE was administered before the SIS, compared to Sites 1 and 3, where the SIS was administered before the MMSE. However, this difference between sites is small. Second, a change in the patient’s cognition between the administration of the first test and the second test may have occurred. We attempted to minimize this by excluding patients who received medications that might affect cognition during the study period. Most commonly, these medications included narcotics and antiemetics. Numerous medications may affect cognition in older adults. It is possible that patients received other medications, such as fluoroquinolones, that could affect their cognition,16 but were not excluded. Clinical issues, such as the fluctuating course of delirium, hypoxia, or hypoperfusion, could also have affected cognition. These clinical parameters were not measured as part of this study.

The improved sensitivity of SIS score calculated from the MMSE, compared to the actual measured SIS score, would argue that one of the previous explanations might have contributed to the discrepant performance in our study. However, even the increased sensitivity found when the SIS score was calculated from the components of the MMSE is lower than the preliminary study noted above. This raises the possibility that the sensitivity of the SIS is limited due to a lack of domains tested. The SIS measures only two domains of cognition—temporal orientation and recall. Untested domains may have resulted in cognitive impairment identified by the MMSE, but not by the SIS. Other short screening tests, for instance, include calculation as a domain.

To the best of our knowledge, this study is the first to question the sensitivity of the SIS for the detection of cognitive impairment in older ED patients. The results of this study suggest that the SIS should not be used as a substitute for more complex, lengthy tests (such as the MMSE). However, we believe that the SIS, testing temporal orientation and recall, is quick and easy for EPs to incorporate into their physical examination. It provides an objective measure of cognition, as opposed to the unstructured evaluation of cognition by clinical gestalt (often expressed as “A&Ox3”). Further study of brief screening instruments for cognitive impairment in the ED should incorporate additional domains. For instance, simple calculation could be added to the SIS; this would not likely substantially increase administration times and would not require the use of pens, paper, or scoring sheets.

Limitations

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. Acknowledgments
  9. References

In addition to the potential limitations discussed above, this was a convenience sample, and thus the possibility of selection bias cannot be excluded. More than 20% of the patients we approached refused to participate, which could also introduce bias. The sensitivity of the SIS could have been affected by this bias, if patients with more severe impairment were not enrolled. However, our cognitive impairment rate of 32% is consistent with other ED studies, suggesting that there was not a bias against enrolling patients with more severe impairment. In addition, although the order of test presentation varied among sites, test administration was not randomized at each center, resulting in an increased possibility of learning phenomenon, confounding the study results. The study population was derived from three urban academic EDs, limiting the external validity for nonacademic, nonurban centers. Finally, the MMSE was used as criterion standard for the diagnosis of cognitive impairment, instead of formal structured psychiatric interviews.

Conclusions

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. Acknowledgments
  9. References

In this multicenter study, we found the sensitivity of the SIS for detection of cognitive impairment, measured by the MMSE, to be lower than in other studies. The reasons for this lower sensitivity are unclear. The SIS remains a rapid and simple method for objectively assessing cognition in older ED patients. However, the results of this study suggest that it cannot be used as a substitute for longer cognitive screening tests. Further study may help to further clarify the ideal characteristics of a brief mental status test for ED use.

Acknowledgments

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. Acknowledgments
  9. References

The authors acknowledge the assistance of John Skruck, Felicia Caffery, Anjan Tibrewala, and T. J. Krall in the enrollment of patients.

References

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. Acknowledgments
  9. References