Reliability and validity of the proposed American College of Rheumatology neuropsychological battery for systemic lupus erythematosus

Authors


Abstract

Objective

To examine the reliability and validity of the proposed American College of Rheumatology (ACR) neuropsychological battery for patients with systemic lupus erythematosus (SLE).

Methods

Thirty-one SLE patients with a history of neuropsychiatric symptoms (NPSLE), 22 SLE patients without a history of neuropsychiatric symptoms (non-NPSLE), and 25 healthy controls completed measures of cognition at baseline and after 1 month. The 1-hour proposed ACR-SLE battery was compared with a 4-hour comprehensive battery (CB).

Results

Seven of 12 measures from the ACR-SLE battery were lower in SLE patients compared with controls. Overall agreement between impairment on the ACR-SLE battery and the CB was 90%. This was established using previously defined impairment on the CB and 4 of 12 scores impaired on the ACR-SLE battery. Almost perfect agreement between the 2 batteries was found for non-NPSLE patients and healthy controls (95–96%) and moderate agreement was reported for NPSLE patients (81%). Intraclass correlation coefficients for ACR-SLE tests ranged from 0.40 to 0.90, indicating adequate reliability.

Conclusion

Reliability and validity of the ACR-SLE battery was established in this study. Agreement regarding classification for impairment was almost perfect for non-NPSLE and moderate for the NPSLE patients. The ACR battery is well designed for general classification of cognitive impairment in SLE. However, comprehensive testing may be useful in identifying specific deficits in NPSLE.

INTRODUCTION

Systemic lupus erythematosus (SLE) is a chronic autoimmune disease that is characterized by multisystem involvement and diverse manifestations. A large fraction of patients with SLE demonstrate organic psychiatric and neurologic disorders, indicating central nervous system (CNS) involvement (1, 2). Neuropsychiatric (NP) manifestations in SLE are diverse and may include major manifestations (i.e., stroke syndromes, severe organic brain syndrome, seizures, psychotic episodes, etc.) or minor abnormalities, including mood disorders and less severe cognitive deficits. As expected, SLE subjects with overt NP symptoms demonstrate severe cognitive and psychological abnormalities (3–10). However, even in the absence of a prior history of CNS lupus (non-NPSLE), cognitive dysfunction and mild psychological distress has been identified (3–9, 11–16). The prevalence of cognitive abnormalities across studies has ranged from 14% to 75%. This range of frequencies may reflect differences in subject selection, neuropsychological test administered, and the definition of cognitive impairment.

An ad hoc multidisciplinary committee of 35 members was convened by the American College of Rheumatology (ACR) Research Committee in April of 1997 for the purpose of developing standard nomenclature for NPSLE (17). At this meeting, recommendations for diagnostic testing were produced across several disciplines, including neuropsychology. Case definitions for 19 neuropsychiatric syndromes were developed and diagnostic agreement was empirically evaluated. Cognitive dysfunction was defined as “significant deficits in any or all of the following cognitive functions: complex attention, executive skills (e.g., planning, organizing, sequencing), memory (e.g., learning, recall), visual-spatial processing, language (e.g., verbal fluency), and psychomotor speed.” The ACR committee also proposed a standard battery of neuropsychological tests for use in people with SLE. The committee discussed a number of brief mental status examinations but, due to the high false-negative rates and failure to detect mild cognitive impairment (18, 19), it was thought that they could not substitute for detailed neuropsychological assessment. Due to the limitations imposed by the longer comprehensive batteries (time requirements and financial burden), the committee recommended a brief 1-hour battery. The tests selected have demonstrated decline in SLE patients in prior studies. Deficits in complex attention were pronounced across studies, as well as deficits in learning and recall for verbal and nonverbal material, verbal and nonverbal fluency, complex psychomotor functions, visuospatial skills, and motor dexterity (3–7, 10, 11, 15, 16, 20–27).

At the time the ACR-SLE battery was proposed, interpretation of test results and identification of impairment was to be based on existing normative data for age, education, sex, and ethnic group variables whenever possible. The ACR-SLE battery was primarily designed as a research tool, and clinical tests were more broadly suggested. To date, no studies have examined the statistical properties (reliability and validity) of the ACR-SLE battery in SLE patients (both with and without a history of NPSLE) compared with controls. Classification of cognitive impairment can be obtained through several approaches. Some studies compare patient and control groups on tests to assess statistical decline. Other studies utilize norm-referenced criteria and domain groups to determine if SLE patients are performing in the impaired range. Definitions of impairment frequently refer to 1 or 2 standard deviations below the norm on individual tests or domains. No SLE studies to date have compared classification schemes or established the sensitivity and specificity of the neuropsychological batteries in people with SLE.

To establish the utility of the proposed ACR-SLE battery, the scores and classification schemes will be compared with a larger previously established neuropsychological battery with SLE patients. To date, there is no gold standard battery of tests for SLE patients. The most comprehensive battery available and utilized with this population was selected. Reliability of the specific tests proposed for the ACR-SLE battery has not been established with the SLE population. Reliability data for these tests can be gathered using a test–retest design in both SLE and control groups. This data will provide information relevant to assessing clinical change (28). This study aimed to 1) compare the frequency of neuropsychological impairment in NPSLE patients, non-NPSLE patients, and healthy controls using the proposed ACR-SLE battery, 2) to establish impairment criteria on the ACR-SLE battery based on a larger comprehensive battery (CB), and 3) to determine reliability of the ACR-SLE battery.

SUBJECTS AND METHODS

Subjects.

Participants in this study included 31 SLE patients with a history of neuropsychiatric symptoms due to CNS lupus (NPSLE), 22 SLE patients without a history of CNS lupus (non-NPSLE), and 25 healthy controls. All subjects signed an approved consent form authorized by the Institutional Review Board at the National Jewish Medical and Research Center. The SLE patients were obtained from a pool of SLE patients seen at the National Jewish Medical and Research Center, The University of Colorado Health Sciences Center, and local rheumatology clinics. SLE subjects with possible neurologic damage prior to SLE (head trauma; degenerative, vascular, or metabolic disorder; neoplasm; or toxic exposure), history of major substance abuse, or major psychopathology were excluded from the study.

The patients were screened using a detailed neuromedical interview that took ∼30–45 minutes to administer. First, participants were screened over the telephone to determine eligibility for inclusion in the study. They were asked to provide age, education, and ethnicity. Participants were questioned as to whether they had a history of head injury, major medical illness, mental illness, a diagnosis of a learning disability, neurological disorder, exposure to toxic chemicals, or a substance abuse problem. If subjects answered yes to any of these questions, they were asked to provide details to determine eligibility. During the first session, additional questions regarding the following areas were asked: demographics, education history (grades and history of learning disability), medical history (diagnosis of lupus, major medical illnesses, history of hospitalization, presence of cardiovascular condition, exposure to toxic chemicals), neurologic history (head injury, stroke, encephalitis, meningitis, epilepsy, brain tests), mental health history (history of therapy, psychotropics, diagnosis of mood disorder, current emotional difficulties, prior psychological testing), substance abuse history (alcohol and drug consumption, substance treatment), and vision or hearing problems. As before, subjects that answered yes to any questions were asked to provide details. Subjects were also asked to provide a list of all current medications.

The SLE patients were randomly recruited across clinics and given information from their physicians regarding participation in the study. None of the SLE patients in this study had previously participated in investigations from this author. All SLE patients fulfilled the revised criteria for SLE as defined by the American College of Rheumatology (29).

All subjects were readministered the ACR-SLE battery ∼1 month following baseline. The mean followup time across the groups was 4.8 weeks (SD 1.0) and none of the subjects had major medical or new neuropsychiatric events within this period.

The 25 healthy controls were recruited in the Denver metropolitan area by newspaper advertisements and brochures or fliers. All of the control subjects were screened using the same detailed neuromedical interview described above for SLE patients. None of the controls had any history of learning problems or medical neuropsychiatric diagnoses, and only 1 of the controls was taking a prescribed medication (levothyroxine).

Subject demographics and health characteristics can be found in Table 1. There were 30 female and 1 male NPSLE patients, 20 female and 2 male non-NPSLE patients, and 23 female and 2 male healthy control participants. As indicated in Table 1, the groups did not significantly differ on age, education level, sex distribution, or race/ethnicity. The estimated premorbid full-scale intelligence quotient obtained from the National Adult Reading Test (30) was 110 (SD 7.2) in the NPSLE group, 108.2 (SD 6.3) in the non-NPSLE group, and 112.7 (SD 4.8) in the control group; scores were marginally different across groups (P = 0.048).

Table 1. Demographics and health characteristics for NPSLE, non-NPSLE, and healthy control groups*
 NPSLE (n = 31)Non-NPSLE (n = 22)Controls (n = 25)P
  • *

    NPSLE = systemic lupus erythematosus with neuropsychiatric symptoms; non-NPSLE = systemic lupus erythematosus without neuropsychiatric symptoms; AA = African American; W = white; H = Hispanic; SLEDAI = Systemic Lupus Erythematosus Disease Activity Index; NA = not applicable.

Demographics    
 Age, mean ± SD years44.8 ± 11.144.4 ± 12.643.5 ± 11.50.91
 Education, mean ± SD years15.2 ± 2.414.4 ± 2.715.4 ± 2.00.31
 Sex, female/male30/120/223/20.61
 Ethnicity, AA/W/H2/27/22/18/22/23/00.69
Health characteristics    
 Disease duration, mean ± SD months102.3 ± 93.676.5 ± 77.6NA0.20
 SLEDAI6.6 ± 5.46.1 ± 4.8NA0.74
 Prednisone dosage, mg/day7.28 ± 9.516.06 ± 9.81NA0.65

SLE disease activity was measured with the Systemic Lupus Disease Activity Index (SLEDAI) and was obtained from the patient's rheumatologist at the time of enrollment. As indicated in Table 1, the SLEDAI was similar across the 2 SLE groups, and a mean score of 6 would suggest mild disease activity (31). The SLE groups did not differ in terms of length of disease, with the NPSLE group at 8.5 years duration and the non-NPSLE group at 6.4 years duration. At the time patients were studied, 61% of the NPSLE and 45% of the non-NPSLE patients were taking prednisone (P = 0.25). The mean dosage of prednisone was similar across the NPSLE and non-NPSLE groups (see Table 1). Additional medications by the NPSLE and non-NPSLE groups at the time of the study included nonsteroidal antiinflammatory drugs (52% and 36%), antidepressants (48% and 50%), thyroid medication (61% and 14%), antihypertensives (45% each), estrogen (13% and 45%), gastrointestinal medications (45% and 41%), and opioid analgesics (29% and 9%).

A checklist of current and prior neuropsychiatric symptoms (using the ACR proposed classifications excluding cognitive dysfunction) was requested from the primary physician to designate non-NPSLE or NPSLE status. Our rheumatology consultant (SW) reviewed all the neuropsychiatric checklists and interviews to assure accurate classification. Of the 31 patients in the NPSLE group, physicians indicated that 15 (48%) had a mood disorder; 14 (45%) headache; 12 (39%) cognitive disorder; 9 (29%) seizure disorder; 2 (6%) stroke, movement disorders, or psychotic disorders; and 1 (3%) aseptic meningitis. Physicians reported that 39% of their NPSLE patients had a cognitive disorder; however, this physician-rated neuropsychiatric symptom was not utilized to create the NPSLE group given the nature of the study. Of the SLE patients who had only mood disorder marked on the checklist, followup inquires were performed by the rheumatologist to assure that active SLE was felt to be the cause of the reported mood disorder and that it improved with immunosuppressive therapy. Most of the patients had >1 neuropsychiatric symptom listed. As indicated previously, none of the healthy controls had any presence or history of neuropsychiatric disorders.

Measures.

The analyses for this study utilized the following neuropsychological tests and were administered by technicians trained in standardized assessment procedures.

The ACR neuropsychology battery for SLE (ACR-SLE).

The Revised Wechsler Adult Intelligence Scale (WAIS-R) Digit Symbol Substitution Test (Digit Symbol) (32) requires psychomotor speed, concentration, and graphomotor abilities. The Trail Making Test part B (Trail Making-B) (33) measures psychomotor speed, attention, and cognitive sequencing. The Stroop Color and Word Test (Stroop Color-Word) (34) measures complex attention and shifting of sets by naming color print for words written in different colors. The learning trial (summation of trials 1–5) and short delay “free” scores from the California Verbal Learning Test (CVLT) (35) were selected to measure learning and recall of verbal material. The immediate and 30-minute delayed-recall measure of the Rey-Osterrieth Complex Figure Test (36) was selected to assess visual learning and memory. The Wechsler Adult Intelligence Scale, Third Edition, Letter Number Sequencing (37) was selected as a measure of auditory working memory. The Controlled Oral Word Association Test (letter fluency) and the Animal Naming Test (category fluency) (38) were selected as measures of letter and category verbal fluency by having subjects generate as many words as possible that begin with a specified letter (F, A, or S) within 60 seconds, and to give the names of as many animals as possible within 60 seconds. The Finger Tapping Test (33) was used as a test of simple fine motor speed bilaterally (dominant and nondominant hands).

The comprehensive neuropsychological battery (CB).

A comprehensive battery previously established with SLE was administered (11). The neuropsychological tests evaluate 8 cognitive domains: intelligence (32), attention (39, 40), reasoning (33), learning (35, 41), recall (41), fluency (38, 42), language (43, 44), and perceptual motor (32). Subjects with 2 of 8 cognitive domains below a mean t-score of 40 were classified as cognitively impaired. In our initial study, ∼33% of the non-NPSLE patients and 11% of the controls were classified as impaired based on this criterion (11).

RESULTS

Comparison of ACR-SLE subtests across groups.

The battery includes 10 specific tests. Each test can generate several scores, and we have selected 12 scores to represent the battery. Each score was demographically corrected utilizing available normative data for each test to produce a t-score (35, 45). The mean t-score and SD for each of the selected tests, by group, can be found in Table 2.

Table 2. ACR-SLE battery test scores by group*
 NPSLE (1)Non-NPSLE (2)Controls (3)PPost-hoc analyses
  • *

    ACR-SLE = American College of Rheumatology neuropsychological battery for patients with systemic lupus erythematosus; NPSLE = systemic lupus erythematosus with neuropsychiatric symptoms; non-NPSLE = systemic lupus erythematosus without neuropsychiatric symptoms; Digit Symbol = Wechsler Adult Intelligence Scale (WAIS) Revised Digit Symbol Substitution Test; Letter Number = WAIS Third Edition Letter Number Sequencing; Stroop Color-Word = Stroop Color and Word Test; Trail Making-B = Trail Making Test part B; Letter fluency = Controlled Oral Word Association Test; Category fluency = Animal Naming Test; Verbal learning = California Verbal Learning Test trials 1-5; Verbal recall = California Verbal Learning Test short delay; Nonverbal learning = Rey-Osterrieth Complex Figure Test, immediate recall; Nonverbal recall = Rey-Osterrieth Complex Figure Test, 30-minute delayed recall; Tapping = Finger Tapping Test.

Digit Symbol46.9 ± 10.248.0 ± 9.555.3 ± 8.20.0041,2 < 3
Letter Number46.3 ± 10.652.4 ± 11.450.7 ± 9.70.095 
Stroop Color-Word46.6 ± 9.146.6 ± 7.952.4 ± 7.60.0191 < 3
Trail Making-B47.1 ± 11.447.0 ± 10.857.9 ± 10.6< 0.0011,2 < 3
Letter fluency41.8 ± 9.644.1 ± 8.449.1 ± 9.80.0171 < 3
Category fluency46.6 ± 10.549.2 ± 7.953.5 ± 11.40.0471 < 3
Verbal learning45.0 ± 11.748.3 ± 8.752.0 ± 14.40.169 
Verbal recall47.4 ± 11.550.5 ± 9.550.7 ± 12.20.354 
Nonverbal learning41.4 ± 16.544.5 ± 11.948.0 ± 12.60.230 
Nonverbal recall42.0 ± 15.944.0 ± 11.647.9 ± 11.40.265 
Tapping dominant hand47.9 ± 8.947.1 ± 10.954.8 ± 10.60.0161,2 < 3
Tapping nondominant hand48.6 ± 10.146.1 ± 11.155.2 ± 11.80.0152 < 3

The t-scores were compared among groups for each of the 12 demographically corrected scores using one-way analysis of variance. Scores were significantly different across groups for 7 scores (Digit Symbol, Stroop Color-Word, Trail Making-B, letter fluency, category fluency, and Tapping dominant and nondominant hand). Post-hoc analyses were performed using Bonferroni multiple comparisons procedure with the experimentwise type I error rate at the 5% level. Results illustrated in Table 2 indicate that both the SLE groups differed from healthy controls on Digit Symbol, Trail Making-B, and Tapping dominant hand. Additionally, the NPSLE group differed from controls on Stroop Color-Word, letter fluency, and category fluency scores. The non-NPSLE group differed from controls on Tapping nondominant hand. The 2 SLE groups were not significantly different from each other on any score.

Test scores below a t-score of 40 were impaired for all of the selected test scores on the ACR-SLE. The frequency of impairment on these tests for the combined SLE groups can be found in Figure 1. Figure 1 illustrates that the highest frequency of impairment in the SLE patients was on tests of nonverbal learning and memory (Rey-Osterreith immediate and delay scores) and letter fluency (generating words for letters F, A, and S).

Figure 1.

Frequency of impairment on the American College of Rheumatology neuropsychological battery for patients with systemic lupus erythematosus. Rey Delay = Rey-Osterrieth Delay Recall; Rey Immed = Rey-Osterrieth Immediate Recall; FAS = Controlled Oral Word Association Test to letters F, A, and S; Tap Dom = Finger Tapping Test dominant hand; L/#Seq = Wechsler Adult Intelligence Scale-Third Edition Letter Number Sequencing; Digit Sym = Wechsler Adult Intelligence Scale Revised Digit Symbol Substitution Test; Tap ND = Finger Tapping Test nondominant hand; CVLT Learn = California Verbal Learning Test trials 1–5; Stroop CW = Stroop Color and Word Test; Trails B = Trail Making Test part B; CVLT SD = California Verbal Learning Test short delay.

Comparing impairment classification: ACR-SLE versus CB.

Because no criteria for scoring or interpreting this combination of 10 tests (12 scores) was available, several different methods of determining impairment were examined. First, we attempted to match the overall level of impairment on the comprehensive battery (11) with impairment on the ACR-SLE battery. For the CB, impairment was determined if ≥2 of the 8 cognitive domain t-scores (i.e., attention, learning, reasoning, etc.) were below a score of 40 (see reference 11 for complete description). For the purposes of this study, the impairment cutoff for the ACR-SLE battery was calculated 2 ways, using individual scores below a t-score of 40 and scores below a t-score of 35. Sensitivity and specificity were calculated for the ACR-SLE battery compared with the CB. Kappa statistic (κ) was also calculated as a measure of agreement between the batteries. Using the t-score <40, we found an overall agreement of 90% (sensitivity 86%, specificity 91%, κ = 0.75) with the original CB impairment index using a cutoff ≥4 ACR-SLE tests, a score suggesting almost perfect association (46). As indicated in Figure 2, we found that the decision to set the cutoff at ≥4 of the 12 ACR-SLE battery scores below a t-score of 40 yielded the following: In the NPSLE group, agreement across the 2 batteries was 81% (sensitivity 80%, specificity 81%, κ = 0.61) with 15 (48%) of 31 impaired on the comprehensive battery and 15 (48%) of 31 impaired on the ACR-SLE battery.

Figure 2.

Comparison of American College of Rheumatology Systemic Lupus Erythematosus (ACR-SLE) and Comprehensive Battery Impairment Levels. A, T-score <40 impaired. B, T-score <35 impaired. In both, ACR impaired if ≥ 4 of 12 test scores impaired, and Comprehensive Battery impaired if ≥ 2 of 8 domain scores impaired. Y1Y3 = Impaired ACR and Comprehensive Battery; Y1 N4 = Impaired ACR and not impaired Comprehensive Battery; N2Y3 = Not impaired ACR and impaired Comprehensive Battery; N2N4 = Not impaired and Comprehensive Battery; NPSLE systemic lupus erythematosus with neuropsychiatric symptoms; Non-NPSLE = systemic lupus erythematosus without neuropsychiatric symptoms.

This level suggested moderate agreement between the 2 batteries in the NPSLE group. In the non-NPSLE group, agreement across the 2 batteries was 95% (sensitivity 100%, specificity 94%, κ = 0.86) with 4 (18%) of 22 of the patients impaired on the CB and 5 (23%) of 22 impaired on the ACR-SLE battery. In the control group, agreement across the 2 batteries was 96% (sensitivity 100%, specificity 96%, κ = 0.78) with 2 (8%) of 25 impaired on the CB, and 3 (12%) of 25 impaired on the ACR-SLE battery. Both the non-NPSLE and healthy controls demonstrated substantial agreement between the 2 batteries. Discrepancies between the 2 batteries occurred in 6 (19%) of 31 of NPSLE patients, 1 (5%) of 22 non-NPSLE patients, and 1 (4%) of 25 controls.

We also compared agreement across the batteries utilizing a more stringent cutoff of 35; the overall the agreement was 95% (sensitivity 80%, specificity 97%, κ = 0.77). In the NPSLE group, agreement was 90% (sensitivity 75%, specificity 96%, κ = 0.74) with 8 (26%) of 31 impaired on the comprehensive and 7 of 31 (23%) impaired on ACR-SLE battery. In the non-NPSLE group, overall agreement between the 2 batteries was 100% (sensitivity 100%, specificity 100%, κ = 1.00) with 2 (9%) of 22 patients impaired on both the comprehensive and ACR-SLE batteries. Finally, agreement was 96% (sensitivity not applicable, specificity 96%, κ = not applicable) in the control group between the batteries with none of the subjects impaired on the CB and 1 (4%) of 25 impaired on the ACR-SLE battery.

We examined more closely the individual test scores from the comprehensive battery in the NPSLE group because agreement across the 2 batteries was the lowest in this group. As noted previously, the battery consisted of 8 domains, and within each domain there were multiple tests representative of that particular cognitive function. For example, the attention domain (38, 40) included 3 scores: Digit Vigilance-Time, Digit Vigilance-Error score, and Paced Auditory Serial Addition Test (PASAT). As indicated in Table 3, the NPSLE patients had significantly lower scores than controls on the Category Test, the PASAT, and the Wechsler Adult Intelligence Scale-Revised Block Design test (WAIS-R).

Table 3. Comparison of individual and domain neuropsychology test scores from the comprehensive battery across groups*
VariableNPSLE Mean ± SDNon-NPSLE Mean ± SDControl Mean ± SDPPost-hoc analyses
  • *

    Post-hoc analyses P value = 0.0167. NPSLE = systemic lupus erythematosus with neuropsychiatric symptoms; non-NPSLE = systemic lupus erythematosus without neuropsychiatric symptoms; WAIS-R = Wechsler Adult Intelligence Scale-Revised; IQ = Intelligence Quotient; PASAT = Paced Auditory Addition Test; Figure learning = Learning component of Figure Memory Test; Story learning = Learning component of Story Memory Test; CVLT = California Verbal Learning Test; Figure recall = Delayed component of Figure Memory Test; Story recall = Delayed component of Story Memory Test; Figural fluency = Ruff Figural Fluency Test; Letter fluency = Controlled Oral Word Association Test; Oral-verbal comprehension = Boston Diagnostic Aphasia Examination-Complex Ideational Material Test; Written comprehension = Peabody Individual Achievement Test-reading recognition test.

Intelligence45.2 ± 9.847.7 ± 8.251.5 ± 6.80.02671 < 3
 WAIS-R Verbal IQ43.9 ± 9.947.1 ± 9.549.8 ± 7.70.0590 
 WAIS-R Performance IQ46.5 ± 11.748.2 ± 9.253.2 ± 7.70.04431 < 3
Reasoning46.4 ± 8.448.6 ± 7.155.7 ± 6.20.00011, 2 < 3
 WAIS-R Similarities46.2 ± 10.648.1 ± 8.151.4 ± 9.60.1359 
 Category test45.3 ± 8.648.0 ± 9.852.2 ± 9.30.02231 < 3
 Trail Making-A47.2 ± 12.851.2 ± 13.561.2 ± 9.50.00021, 2 < 3
 Trail Making-B47.1 ± 11.447.0 ± 10.857.9 ± 10.60.00061, 2 < 3
Attention40.8 ± 6.243.7 ± 6.050.0 ± 6.90.00011, 2 < 3
 Digit Vigilance Test-Time48.2 ± 10.347.9 ± 8.654.8 ± 11.60.03021, 2 < 3
 Digit Vigilance Test-Error41.1 ± 8.444.5 ± 11.648.1 ± 12.00.0542 
 PASAT total33.2 ± 13.938.8 ± 12.547.2 ± 9.00.00031 < 3
Learning42.0 ± 7.544.1 ± 5.847.5 ± 7.90.02061 < 3
 Figure learning41.5 ± 11.440.5 ± 8.246.7 ± 13.70.1312 
 Story learning39.4 ± 9.643.5 ± 11.545.2 ± 9.20.0918 
 CVLT trials 1–545.0 ± 11.748.3 ± 8.750.7 ± 2.20.1686 
Recall49.5 ± 6.850.1 ± 5.851.3 ± 7.20.5816 
 Figure recall50.1 ± 9.252.6 ± 7.753.8 ± 6.40.2011 
 Story recall48.9 ± 10.447.5 ± 9.848.9 ± 10.10.8697 
Fluency46.0 ± 7.546.6 ± 5.948.2 ± 6.50.4834 
 Figural fluency50.2 ± 10.649.1 ± 8.447.3 ± 8.80.5056 
 Letter fluency41.8 ± 9.644.1 ± 8.449.1 ± 9.80.01681 < 3
Language45.9 ± 9.447.6 ± 9.448.4 ± 7.00.5278 
 Oral-verbal comprehension43.0 ± 10.247.4 ± 10.447.5 ± 7.90.1524 
 Written comprehension48.7 ± 13.648.0 ± 12.449.4 ± 11.40.9305 
Perception47.4 ± 10.549.6 ± 9.952.6 ± 7.40.1355 
 WAIS-R Block design48.9 ± 11.050.3 ± 12.256.2 ± 10.20.04481 < 3
 WAIS-R Object assembly46.0 ± 11.448.9 ± 10.048.9 ± 7.60.4430 

We also compared the 6 NPSLE patients that were discordant across the 2 batteries with the healthy controls and found significant differences on domain scores of intelligence, reasoning, attention, and perception. Specific tests that differed included the Trail Making-A (P = 0.014), WAIS-R similarities subtest (P = 0.018), WAIS-R block design subtest (P = 0.011), and WAIS-R object assembly subtest (P = 0.025).

Reliability and practice effects for the ACR-SLE battery.

Intraclass correlation coefficients were calculated for the individual tests in the ACR-SLE battery using the test–retest raw score values as shown in Table 4.

Table 4. Intraclass correlation coefficients in ACR-SLE by group*
 NPSLENon-NPSLEControls
  • *

    r < 0.4 indicates poor reproducibility; 0.4 ≤ r < 0.75 indicates fair to good reproducibility; r ≥ 0.75 indicates excellent reproducibility. ACR-SLE = American College of Rheumatology neuropsychological battery for patients with systemic lupus erythematosus; NPSLE = systemic lupus erythematosus with neuropsychiatric symptoms; non-NPSLE = systemic lupus erythematosus without neuropsychiatric symptoms; Digit Symbol = Wechsler Adult Intelligence Scale (WAIS) Revised Digit Symbol Substitution Test; Letter Number = WAIS Third Edition Letter Number Sequencing; Stroop Color-Word = Stroop Color and Word Test; Trail Making-B = Trail Making Test part B; Letter fluency = Controlled Oral Word Association Test; Category fluency = Animal Naming Test; Verbal learning = California Verbal Learning Test trials 1-5; Verbal recall = California Verbal Learning Test short delay; Nonverbal learning = Rey-Osterrieth Complex Figure Test, immediate recall; Nonverbal recall = Rey-Osterrieth Complex Figure Test, 30-minute delayed recall; Tapping = Finger Tapping Test.

Digit Symbol0.800.880.89
Letter Number0.630.400.67
Stroop Color-Word0.790.580.84
Trail Making-B0.720.830.57
Letter fluency0.730.810.84
Category fluency0.730.600.49
Verbal learning0.730.600.51
Verbal recall0.670.560.25
Nonverbal learning0.730.750.49
Nonverbal recall0.720.860.51
Tapping dominant hand0.820.900.90
Tapping nondominant hand0.810.850.88

Overall, the reproducibility was fair to excellent for a majority of the tests across all the groups, except for the CVLT short delay in the healthy controls (r = 0.25).

Significant practice effects were found for all groups on Digit Symbol, letter fluency, CVLT trials 1–5, and Rey-Osterreich immediate and delayed recall (Table 5). The non-NPSLE group had an additional significant practice effect for Trail Making-B. The NPSLE group had additional significant practice effects for the Letter Number Sequencing, the Stroop Color-Word, CVLT short delayed recall, and Tapping dominant hand. However, these practice effects are not significantly different when compared across groups.

Table 5. Change in ACR-SLE scores time 2 to time 1*
 NPSLENon-NPSLEControls
Change (time 2–1)PChange (time 2–1)PChange (time 2–1)P
  • *

    For abbreviations and complete test names, see Table 4.

Digit Symbol3.30.01413.10.04822.90.0496
Letter Number1.00.02500.60.27340.30.5790
Stroop Color-Word3.80.00101.70.20642.60.0362
Trail Making-B0.70.8580−15.80.0012−3.80.3907
Letter fluency3.80.00437.60.00015.60.0003
Category fluency0.80.33751.20.2555−0.40.7109
Verbal learning6.20.00017.50.00017.00.0001
Verbal recall1.50.00080.90.09190.90.0563
Nonverbal learning4.10.00015.10.00014.70.0001
Nonverbal recall4.50.00014.60.00015.00.0001
Tapping dominant hand1.60.0319−0.10.90121.10.1952
Tapping nondominant hand1.20.1071−0.30.70350.90.2989

DISCUSSION

This study indicates acceptable validity and reliability for the proposed 1-hour ACR battery for SLE patients. Measures of visuomotor speed, complex visual attention, motor sequencing, letter fluency, category fluency, and motor speed were lower in SLE patients compared with controls. In contrast to prior studies, there were no differences between the NPSLE and non-NPSLE patients in cognitive scores. This may be due to the large number of patients who had a prior history of mild symptoms of neuropsychiatric lupus, such as mood disorders and headache. Overall, the cognitive functions that showed decline in this study of SLE patients were consistent with prior SLE studies that demonstrate deficits in visuospatial processing and speed (6, 13, 24, 47, 48), verbal fluency (3, 11, 25), and attention (5, 6, 22, 48). We did not find a significant difference in immediate and delayed recall scores for verbal and nonverbal material using the group comparison; however, 40% or more of the SLE patients were impaired in immediate and delayed recall of visual material (using demographically corrected normative data). Additionally, >40% of the SLE patients were impaired in verbal fluency to letter cues, and 20% or more of the SLE patients were impaired in bilateral motor functions, working memory, visuospatial speed, category fluency, and learning of verbal material. Overall, these findings suggest that the ACR-SLE 1-hour battery does consistently identify cognitive impairment in both NPSLE and non-NPSLE patients.

Our analysis indicated substantial sensitivity and specificity of this brief 1-hour ACR-SLE battery in healthy controls and non-NPSLE patients compared with a larger 4-hour comprehensive battery. However, for SLE patients with a history of neuropsychiatric lupus, there was a drop in the overall agreement. This may be due to the multiple and complex presentation of these patients and the limitations of a brief battery with only 12 test scores to detect a wide range of cognitive domains. For example, 30% of the NPSLE patients had a seizure disorder, a condition that may affect brain function in multiple dimensions. NPSLE patients in our study were also impaired compared with controls on measures of visuoconstructive ability, higher-level reasoning, and information processing, areas not strongly represented in the ACR-SLE battery. Therefore, although the short battery is sufficient for identifying global impairment in NPSLE, a more comprehensive battery of tests may be more appropriate for various clinical evaluations and investigative designs. It should be noted that the larger CB did not have a pure motor domain, whereas the ACR-SLE battery had 2 scores (that were quite impaired) in this area. Although this is not likely the main reason for the lower agreement across the 2 batteries in the NPSLE group, it may have contributed to the discrepancy. Continued statistical analysis with other large comprehensive batteries will be useful in improving agreement and providing convergent validity for the ACR battery.

In this study, 48% of the NPSLE patients were impaired on the ACR-SLE battery, and 50% were impaired on the CB. In prior studies, rates of cognitive impairment in NPSLE have been higher than 50% (3, 5, 13, 21). However, other studies suggest a rate between 3% and 40% (7, 9) and between 30% and 40% (6, 8). In non-NPSLE, the rates of impairment are typically lower than the NPSLE patient group. In our study, 23% in the non-NPSLE were impaired on the SLE-ACR and 18% were impaired on the CB. This is lower than some studies that demonstrate impairment in 40–50% of these patients (3, 5, 13, 47) but consistent with other studies reporting 20–30% (6–9, 11). Rates of impairment in other healthy controls are typically ∼10%; in our study, the rate was 8% for the brief battery and 12% for the longer battery. This would further suggest that a cutoff score ≥4 impaired scores of 12 on the ACR-SLE battery is acceptable. Indices based on 2 or 3 tests impaired in brief batteries may inflate impairment levels in SLE patients. Continued analysis across batteries will be required to establish true prevalence rates.

Reliability for the individual ACR-SLE battery test scores was fairly high on a majority of the tests. Results indicated that reliability was excellent for visuospatial speed and motor function and good for a majority of the other tests. However, verbal recall (CVLT short delay) was in the poor range for the control group, and a measure of working memory (Letter Number Sequencing) was marginal for SLE. Interestingly, significant practice effects (i.e., improved performance due to repeated administration) were found for all groups across all the tests, except category fluency and Tapping nondominant hand. This indicates that caution is required when retesting SLE patients over short intervals and that the reliability coefficients produced in this article may facilitate accurate identification of clinical change over time when computing reliable change coefficients (a statistical approach to quantifying change in a test score that contains measurement error) (28).

There are aspects in the design of our study that limit our conclusions and warrant further investigation prior to establishing the ACR-SLE battery as a gold standard for evaluations. First, for the ACR-SLE selected battery, we utilized only 12 of the multiple available test scores available from the 10 tests. We attempted to use only scores with adequate normative data from which to evaluate impairment. This may have minimized available data. For example, there was a visuoconstructive element to the Rey-Osterreith test (i.e., a task that primarily assesses visual-motor constructive abilities with no memory component); however, standardized normative interpretive data was not available so it was not included as one of the test scores. Second, we utilized our own comprehensive battery (11), and this approach was very conservative compared with other SLE studies in defining impairment. For example, multiple tests were utilized to derive a mean t-score for each of 8 domains. Therefore, the range of deficits was likely minimized by averaging multiple scores within each domain. Additionally, we did not have a motor-only domain score. These domains, such as attention and reasoning, include tests that are qualitatively specified and could be considered in other domains. For example, the Trail Making Test could be considered complex attention as well as a reasoning/sequential task. This may have reduced the overall agreement between the 2 batteries, particularly in the NPSLE group.

The use of cutoff scores has significant clinical limitations, particularly with a brief battery such as the ACR-SLE battery. This cutoff score detects cognitive impairment but not severity. To detect differences between mild, moderate, and severe cognitive impairment, a more refined scoring method, requiring level of performance across each test, needs to be considered individually. In some cases, additional testing will be required. Finally, there are many methodologic difficulties in assigning or classifying SLE patients as having neuropsychiatric disease or not. As mentioned above, we have some preliminary data (49) that notes discordant findings between a physician's report of major mood disorders compared with standardized psychiatric evaluation. To complicate matters, cognitive disorder is now classified as a “neuropsychiatric” syndrome in recent nomenclature (17), thus suggesting that there are many patients who may initially have been in the non-NPSLE group due to absence of their overt NP manifestations, but could be classified as having NPSLE. In our particular study, the patients classified with NPSLE had not only a history of major disorders, such as stroke and seizures, but also a variety of less severe abnormalities, such as mood disorders and headaches. In a larger sample it may be useful to separate these categories, particularly because it is often difficult to determine if the mood disorder or headache is due to SLE or another cause, such as reactive depression. Additionally, our methods did not differentiate active versus inactive neuropsychiatric disease. In review, 7 (22%) of 31 of the NPSLE patients had items from the SLEADI indicating active seizure, headache, and organic brain syndrome disorders. Due to the expansion of symptoms fitting neuropsychiatric disorders, future studies would benefit from additional ratings of severity and activity across multiple neuropsychiatric features. Finally, our SLE subjects' characteristics showed limited variability (i.e., mostly white, highly educated) and continued analysis across various age, ethnic, and educational levels should be considered.

In summary, the use of the ACR-SLE neuropsychological battery is overall valid and reliable, and will likely improve the accessibility of neuropsychological testing to SLE patients by providing a short battery to classify impairment. Practice effects for some of the selected tests were high and may require more stringent interpretation when assessing change over time.

Ancillary