Validation of the Pediatric Automated Neuropsychological Assessment Metrics in childhood-onset systemic lupus erythematosus

Authors


  • The contents of this article are solely the responsibility of the authors and do not necessarily represent the official views of the NIH.

Abstract

Objective

To evaluate the reproducibility and validity of the Pediatric Automated Neuropsychological Assessment Metrics (Ped-ANAM) when used in childhood-onset systemic lupus erythematosus (cSLE).

Methods

Forty children with cSLE and 40 matched controls were followed for up to 18 months. Formal neuropsychological testing at baseline was repeated after 18 months of followup; overall cognitive performance and domain-specific cognition (attention, working memory, processing speed, and visuoconstructional ability) were measured and categorized as normal cognition, mild/moderate, or moderate/severe impairment. The 10 Ped-ANAM subtests were completed every 6 months and twice at baseline. Ped-ANAM performance was based on accuracy (AC), mean time to correct response (MNc), throughput, and coefficient of variation of the time required for a correct response (CVc) as a measure of response consistency.

Results

Particularly, MNc scores demonstrated moderate to substantial reproducibility (intraclass correlation coefficients 0.47–0.80). Means of select Ped-ANAM scores (MNc, AC, CVc) differed significantly between children with different levels of cognitive performance and allowed for the detection of moderate or severe cognitive impairment with 100% sensitivity and 86% specificity. Six Ped-ANAM subtests significantly correlated with the change in overall cognitive function in cSLE (baseline versus 18 months; Spearman's correlation coefficient >0.4, P < 0.05; n = 24).

Conclusion

The Ped-ANAM has moderate to substantial reproducibility, criterion and construct validity, and may be responsive to change in cSLE. Additional research is required to confirm the outstanding accuracy of the Ped-ANAM in identifying cognitive impairment, as well as its usefulness in detecting clinically relevant changes in cognition over time.

INTRODUCTION

Like adults, children with systemic lupus erythematosus (cSLE) frequently report cognitive problems, and several studies have documented significant cognitive deficits with traditional neuropsychological test batteries (1, 2). Most studies find neurocognitive dysfunction (NCD) on tests measuring attention or concentration, cognitive flexibility, free recall memory, visuoconstructional ability, and speed of information processing in a substantial subgroup of cSLE, suggesting the presence of a subcortical cognitive syndrome (3). NCD may represent active neuropsychiatric lupus. When present, NCD has detrimental effects on patient quality of life (4), and thus constitutes an important disease feature to consider in the medical management of cSLE.

The detection of NCD with cSLE is typically made by formal neuropsychological testing (1, 5), but this is costly, time intensive, not always readily available, and requires specialized advanced training. In recent years, computer-administered tests have been explored in patients with various diseases as more cost-effective screening tools of NCD (6). One of these is the Automated Neuropsychological Assessment Metrics (ANAM), designed as a library of automated tasks assessing various aspects of cognitive functioning (7).

The ANAM has been found to be well-suited to screen the cognitive abilities of adults with SLE (3, 8). Similarly, our group reported on the potential usefulness of the adaptation of the ANAM for pediatric use (Ped-ANAM) (9) to screen for cSLE-associated NCD in a small cross-sectional pilot study (10). Building on our promising initial results, our objectives for this case–control study were to more thoroughly document the utility of the Ped-ANAM in this high-risk population; specifically, we aimed to: 1) assess the feasibility of administration by nonneuropsychologist clinic staff, 2) document the short-term reproducibility of the instrument, 3) test its concurrent and criterion validity, and 4) test its responsiveness to change, relative to more traditional neuropsychological tests.

Significance & Innovation

  • This study supports the utility of the Pediatric Automated Neuropsychological Assessment Metrics (Ped-ANAM) as a screening tool for neurocognitive dysfunction in childhood-onset systemic lupus erythematosus.

  • The Ped-ANAM is a feasible screening device with good reproducibility and criterion and concurrent validity in this population.

  • Additionally, neurocognitive dysfunction can be determined with high accuracy.

  • Initial evidence of the responsiveness to change of the Ped-ANAM in cognitive ability over time is provided.

PATIENTS AND METHODS

Forty patients with cSLE (11) were asked to each identify a friend who was within 1 year of his/her age, of the same sex, and in the same school grade. This “best friend approach” to selecting control populations has been shown to result in good case–control matches on sociodemographic variables (12). Controls had to be healthy, without known structural brain abnormalities, or known NCD. No potential controls needed to be excluded from participation by these criteria. The patients' medical records were reviewed for cSLE-relevant parameters, and additional information about the study population is provided elsewhere (13). To study the reproducibility (also known as test–retest reliability) of the Ped-ANAM, study participants completed the test twice during the first study visit. For use as external standard, participants also completed formal neuropsychological tests at the same visit. Finally, both the Ped-ANAM and neuropsychological tests were repeated 18 months later to address responsiveness to change.

Measures.

Ped-ANAM.

The Ped-ANAM has been adapted from the traditional ANAM (14, 15) for use in children ages ≥10 years (16). Together, the Ped-ANAM subtests measure sustained concentration and attention, spatial processing, cognitive processing efficiency, verbal reasoning, learning, recall, and working memory. A full description of the Ped-ANAM subtests is published elsewhere (10) and in Supplementary Table 1 (available in the online version of this article at http://onlinelibrary.wiley.com/doi/10.1002/acr.21835/abstract).

Performance on each of the Ped-ANAM subtests can be gauged by 4 scores: 1) accuracy defined as the percentage of correct responses (AC); 2) mean reaction time for correct responses or speed (MNc; in seconds); 3) throughput (TP), which is considered a measure of effectiveness or cognitive efficiency and is a combination of reaction time and accuracy (17); and 4) coefficient of variation of reaction time for correct responses (CVc; CVc = SD of MNc/MNc), which reflects the consistency of a test taker's response speed within a given subtest. For the Simple Reaction Time subtest, only the MNc score was calculated as this subtest only allows for correct responses.

When taking the Ped-ANAM, the Simple Reaction Time subtest is completed at the beginning and again at the end of the testing session, with the MNc calculated as the average across both subtests. Differences in the performance of the Simple Reaction Time subtest over a given Ped-ANAM session (beginning versus end of session) can be used to assess the change in speed of sensorimotor processing or participant fatigue when taking the Ped-ANAM (18). Higher scores at the end of the session than at the beginning indicate a decline in overall speed of sensorimotor processing. Higher AC and TP scores, and lower MNc and CVc scores, indicate better cognitive performance. The TP variable has been widely used in ANAM research given that it is sensitive to cognitive performance, incorporates speed and accuracy in one variable, and more closely conforms to a normal distribution than other variables (8, 18–20).

Formal neuropsychological testing battery and definition of NCD.

Formal neuropsychological testing was performed by trained psychometricians, using a standardized neuropsychological battery for cSLE, with details provided elsewhere (5). The battery assesses working memory, psychomotor speed, attention, and visuoconstructional ability; these are cognitive domains that have been found to be particularly affected by cSLE in prior research (Table 1). All measures included in this battery of neuropsychological tests are well validated and provide age-normed scores compared to large, demographically diverse normative samples (5).

Table 1. Tests used to define neurocognitive dysfunction
Domain/measureSourceDescription
Working memory  
 Digit spanAge-appropriate Wechsler Intelligence Scale (38, 39)Ability to repeat back in order, or in a resequenced order, increasingly difficult strings of numbers
 Letter-number sequencingAge-appropriate Wechsler Intelligence Scale (38, 39)Ability to mentally resequence a series of letters and numbers before repeating them back
Psychomotor speed  
 CodingAge-appropriate Wechsler Intelligence Scale (38, 39)Test takers “decode” and transcribe a series of symbols as quickly as possible
 Symbol searchAge-appropriate Wechsler Intelligence Scale (38, 39)Score reflects speed and accuracy of test taker's visual search for matches in rows of symbols
Attention  
 Hit reaction time SEConners Continuous Performance Test II (40)On a 15-minute boring task, the variability in reaction time to specific letters flashing on screen
 Inhibition vs. color naming  scoreDelis-Kaplan Executive Functioning System (41)Relative ability to focus on ink color in which a conflicting color word is printed (e.g., “blue” written in red ink)
Visuoconstructive abilities  
 Block designWechsler Abbreviated Scales of Intelligence (42)Ability to efficiently reproduce colored line drawings using blocks with sides that have varying patterns
 Block countingKaufman Assessment Battery for Children (43, 44)Ability to mentally represent the volume of a 3-dimensional block construction printed in 2-dimensional space

To categorize levels of cognitive function, participants' age-normed scores were converted to a common Z score metric (mean = 0 and SD = 1 for a normative healthy population), with higher scores reflecting better performance. The mean Z scores within each cognitive domain were then averaged to yield a composite domain score, with 4 composite domain scores overall. Finally, in the absence of a generally accepted definition for NCD (21), 3 levels of cognitive ability were defined: 1) normal cognition/no NCD if all Z scores are −1 or more; 2) mild to moderate NCD if 1 or 2 domain Z scores are less than −1, or 1 domain Z score is −2 or less; and 3) moderate to severe NCD if more than 1 domain Z score is less than −2, or more than 2 domain Z scores are less than −1.

Statistics.

Demographic and clinical characteristics were summarized by means and SDs for numerical variables and frequency (%) for categorical variables. We calculated intraclass correlation coefficients (ICCs) to assess the reproducibility of the Ped-ANAM subtests. ICCs can be interpreted as follows: ICC <0.4 for poor agreement; ICC ≥0.4–0.75 for fair to good agreement; and ICC >0.75 for substantial to excellent agreement (22).

As previously suggested (18), performance fatigue was examined by comparing the difference of the MNc score on the Simple Reaction Time subtest when administered in the beginning as compared to the end of a Ped-ANAM session. Both Spearman's and Pearson's correlation coefficients were calculated to assess relationships between Ped-ANAM scores and NCD domain and overall Z scores. Results from these 2 methods were similar, thus only Spearman's correlation coefficients (r) are reported. Absolute values of r can be interpreted as unrelated; weak, moderate, or strongly correlated for values <0.2; 0.2–0.39, 0.4–0.59, or >0.6, respectively (23). To establish criterion validity, fixed-effect models were used to determine the associations of Ped-ANAM measures with cognitive function categories (normal cognition, mild/moderate NCD, and moderate/severe NCD); means were compared post hoc between NCD groups under the fixed-effect model framework and adjusted for multiple comparisons using Tukey's method. The responsiveness of the Ped-ANAM to cognitive change (based on change as measured by the traditional cognitive tests) was assessed by determining the correlation between the change in the cognitive function (domain Z scores) and the change of the Ped-ANAM scores per subtest.

Both unadjusted and adjusted (after adding demographics and clinical characteristics as controlling covariate) fixed-effect models were considered in computation. Results from adjusted models are not shown as they were similar to those of unadjusted models. In the prediction analyses, moderate/severe NCD was predicted by Ped-ANAM measures using multivariate logistical regression models. Stepwise selection methods were used in the logistical regression models to select Ped-ANAM scores. A propensity score was calculated from each of the logistical models and considered in receiver operating characteristics (ROC) curve analysis to predict moderate/severe NCD. The area under the ROC curve (AUC) was calculated, and sensitivity and specificity determined under a preferred threshold approach. Values of the AUC can be interpreted as outstanding, excellent, good, fair, and poor performance in predicting NCD, for values of 1.0–0.91, 0.81–0.90, 0.71–0.8, 0.61–0.7, and <0.6, respectively (24).

As an alternative approach, classification and regression tree (CART) model analysis (25) was done and a CART score developed from the final nodes of the CART model tested in ROC curve analysis. CART analysis was performed using SYSTAT software, version 11.0, and other statistical analyses were performed using SAS software, version 9.3, with ROC curves plotted using Splus software, version 6.2 (Insightful Corporation). P values less than 0.05 were considered statistically significant.

RESULTS

Study participants and formal neuropsychological testing.

English was the native language for the 40 cSLE patients and 40 controls matched for sex, school grade, and age (within 1 year of age of index patient with cSLE), with sociodemographic details and information about cSLE status provided elsewhere (13) and online (Supplementary Table 2, available in the online version of this article at http://onlinelibrary.wiley.com/doi/10.1002/acr.21835/abstract).

Briefly, 45% of the cSLE patients were African American, and 85% of the study participants were female. Besides the mean ± SD age of the cSLE group being somewhat higher than that of the control group (14.8 ± 2.3 years versus 13.98 ± 3.2 years; P = 0.03), the groups were comparable on sociodemographic factors as measured by the maternal education level and family income. NCD was identified in both controls and the cSLE group, with a trend towards more pronounced NCD severity in the latter. At baseline, normal cognition, mild NCD, or moderate/severe NCD were noted, respectively, in 60% (n = 24), 35% (n = 14), and 5% (n = 2) of the controls as compared to 62.5% (n = 25), 25% (n = 10), and 12.5% (n = 5) of the cSLE patients. Daily prednisone was prescribed to 78% (31 of 40) of the patients with cSLE. Disease activity as measured by the Systemic Lupus Erythematosus Disease Activity Index (mean ± SD 4.9 ± 4.4) and British Isles Lupus Assessment Group 2004 (mean ± SD 3.0 ± 3.8) indices (26) was in the mild to moderate range; 18-month followup data were available for 24 of the 40 cSLE participants.

Feasibility.

None of the participants had difficulties in understanding the Ped-ANAM instructions. Each administration took 35–55 minutes. There were rare technical problems (nonresponsiveness of the software due to repetitive triggering of the mouse in very short time intervals) in 4 patients, requiring the intervention of clinical research personnel.

TP scores.

We found the TP score to be problematic as a measure of cognitive function due to our observation of atypical response patterns in some participants, especially those with NCD, who responded to test items with unusually fast reaction times (i.e., substantially faster than the mean of the group) with correspondingly very low accuracy (often at chance levels), suggesting that these participants responded quickly without actually attending to the cognitive demands of the task. Historically, TP has been shown to be more strongly weighted toward reaction time than accuracy (given the inherent greater range of variability in the reaction time variable compared to accuracy). Therefore, in this study, the TP score of more cognitively complex tests tended to give too much “credit” or to overestimate cognitive functioning in participants with a quick but inaccurate response style.

Reproducibility.

The reproducibility of the Ped-ANAM subtests was highest for subtests of higher cognitive complexity and for the MNc score (Table 2). The reproducibility (MNc scores) of the Ped-ANAM ranged from fair to excellent, but for the most part was good to substantial. AC scores showed fair to good agreement for some subtests (Code Substitution, Matching to Sample, Matching Grids, and Sternberg Memory Search) but showed substantial to excellent consistency for the Continuous Performance Test. ICCs for the derived performance parameter CVc varied widely and ranged from poor (Procedural Reaction Time) to good (Continuous Performance Test).

Table 2. Reproducibility of the Ped-ANAM scores*
Ped-ANAM subtestsPercentage of correct responsesMean reaction time for correct response (MNc)Consistency (CVc = SD of MNc/MNc)
  • *

    Values are the intraclass correlation coefficient (95% confidence interval) adjusted for multiple comparisons using a Tukey's method. Ped-ANAM = Pediatric Automated Neuropsychological Assessment Metrics; CVc = coefficient of variation of the time required for a correct response; NE = not estimable.

Code Substitution Delayed0.38 (0.21–0.55)0.58 (0.44–0.71)0.21 (0.02–0.40)
Code Substitution0.44 (0.27–0.61)0.80 (0.73–0.88)0.43 (0.27–0.60)
Continuous Performance Test0.78 (0.70–0.86)0.71 (0.61–0.81)0.68 (0.57–0.79)
Logical Relations0.23 (0.03–0.43)0.77 (0.69–0.86)0.15 (0.00–0.35)
Matching to Sample0.52 (0.37–0.67)0.64 (0.52–0.76)0.57 (0.42–0.72)
Matching Grids0.43 (0.25–0.60)0.77 (0.68–0.85)0.38 (0.21–0.56)
Mathematical Processing0.25 (0.05–0.45)0.91 (0.87–0.95)0.62 (0.49–0.75)
Procedural Reaction Time0.10 (0.00–0.31)0.47 (0.32–0.62)0.16 (0.00–0.38)
Spatial Processing0.26 (0.03–0.49)0.71 (0.61–0.82)0.34 (0.17–0.51)
Sternberg Memory Search0.55 (0.41–0.69)0.52 (0.38–0.67)0.27 (0.10–0.45)
Simple Reaction TimeNE0.68 (0.57–0.79)NE

Construct validity.

We expected participants with normal cognition to perform better on the Ped-ANAM compared to those with NCD, especially if NCD was more pronounced. There were statistically significant differences in scores (AC, MNc, and CVc) on several Ped-ANAM subtests, particularly in contrasting the performance of the moderate/severe NCD group with the normal cognition and mild/moderate NCD groups (Table 3). Interestingly, when visually examining group means for the MNc score, the moderate/severe NCD group tended to respond the fastest. We believe this is due to the earlier described tendency of some participants to respond quickly without attending fully to the cognitive demands of the task. This possibility is supported by the observation of lower AC scores in this group compared to the other 2 groups and the finding that the moderate/severe NCD group performed significantly slower than the other 2 groups on Simple Reaction Time subtests, which do not present significant cognitive demands (P [MNc] < 0.021 for all).

Table 3. Differences in Ped-ANAM performance with differences in cognitive function as measured by formal neuropsychological assessment*
Ped-ANAM scores/subtestsMean ± SEP
Normal cognition (I)Mild NCD (II)Moderate/severe NCD (III)I vs. III vs. IIIII vs. III
  • *

    Ped-ANAM = Pediatric Automated Neuropsychological Assessment Metrics; NCD = neurocognitive dysfunction; AC = percentage of correct responses per subtest; NS = not significant; CVc = coefficient of variation of reaction time for correct responses; MNc = mean reaction time for correct response (seconds).

  • NCD categories are defined as follows based on Z scores of the standardized tests completed for the formal neuropsycological testing in 4 functional domains (attention/executive function; processing speed; visuoconstructional memory; and working memory). Normal if all Z scores are −1 or more; mild to moderate NCD if 1 or 2 domain Z scores are −1 or 1 domain Z score is −2 or less; and severe NCD if more than 1 domain Z score is less than −2 or more than 2 domain Z scores are less than −1.

  • P values are adjusted for multiple comparisons using Tukey's method.

AC      
 Code Substitution Delayed79.61 ± 1.8277.91 ± 2.5464.27 ± 5.19NS0.0180.050
 Code Substitution96.16 ± 0.5396.02 ± 0.7489.86 ± 1.51NS0.0010.001
 Continuous Performance Test83.67 ± 2.7978.20 ± 3.9859.49 ± 7.97NS0.015NS
 Logical Relations95.56 ± 0.7995.87 ± 1.1187.06 ± 2.26NS0.0020.002
 Matching to Sample85.00 ± 2.2980.52 ± 3.2163.00 ± 6.56NS0.0060.049
 Matching Grids92.65 ± 1.2792.90 ± 1.7785.83 ± 3.62NSNSNS
 Mathematical Processing90.00 ± 0.8390.00 ± 1.1686.20 ± 2.37NSNSNS
 Procedural Reaction Time96.07 ± 0.7394.24 ± 1.0393.33 ± 2.10NSNSNS
 Spatial Processing91.99 ± 1.1390.40 ± 1.5874.81 ± 3.23NS0.0000.000
 Sternberg Memory Search92.08 ± 1.1490.92 ± 1.5987.78 ± 3.25NSNSNS
CVc (SD of MNc/MNc)      
 Code Substitution Delayed0.55 ± 0.030.58 ± 0.040.76 ± 0.09NSNSNS
 Code Substitution0.34 ± 0.010.36 ± 0.020.42 ± 0.04NSNSNS
 Continuous Performance Test0.32 ± 0.010.36 ± 0.020.60 ± 0.04NS0.0000.000
 Logical Relations0.30 ± 0.010.32 ± 0.020.40 ± 0.04NS0.019NS
 Matching to Sample0.46 ± 0.020.45 ± 0.030.71 ± 0.06NS0.0010.001
 Matching Grids0.28 ± 0.010.34 ± 0.020.38 ± 0.040.0330.048NS
 Mathematical Processing0.36 ± 0.020.42 ± 0.030.48 ± 0.06NSNSNS
 Procedural Reaction Time0.29 ± 0.020.29 ± 0.020.41 ± 0.05NS0.050NS
 Spatial Processing0.36 ± 0.020.40 ± 0.020.48 ± 0.05NS0.030NS
 Sternberg Memory Search0.40 ± 0.030.51 ± 0.040.36 ± 0.08NSNSNS
MNc      
 Code Substitution Delayed1.22 ± 0.051.29 ± 0.070.72 ± 0.15NS0.0060.003
 Code Substitution1.14 ± 0.041.25 ± 0.060.98 ± 0.12NSNSNS
 Continuous Performance Test0.59 ± 0.020.63 ± 0.030.55 ± 0.05NSNSNS
 Logical Relations1.54 ± 0.071.72 ± 0.101.46 ± 0.21NSNSNS
 Matching to Sample2.08 ± 0.082.49 ± 0.121.68 ± 0.24NSNS0.009
 Matching Grids1.71 ± 0.082.10 ± 0.111.67 ± 0.220.009NSNS
 Mathematical Processing1.57 ± 0.101.78 ± 0.141.79 ± 0.28NSNSNS
 Procedural Reaction Time0.60 ± 0.020.61 ± 0.020.57 ± 0.05NSNSNS
 Spatial Processing2.09 ± 0.092.44 ± 0.131.71 ± 0.26NSNS0.038
 Sternberg Memory Search0.92 ± 0.041.03 ± 0.060.74 ± 0.11NSNSNS
 Simple Reaction Time0.29 ± 0.010.31 ± 0.020.43 ± 0.03NS0.0010.003

Criterion validity.

In an effort to evaluate criterion validity and determine whether one can limit the number of Ped-ANAM subtests to be completed for the surveillance of NCD, we assessed which particular subtests were especially correlated with the classification of NCD (Table 4). Therefore, we calculated odds ratios using multivariate logistic models (outcome: normal cognition yes/no), while adjusting for age differences between groups. Our results, shown in Figure 1, suggest that the presence of NCD can be accurately predicted by a subset of Ped-ANAM subtests (Spatial Processing, Continuous Performance Test, Matching to Sample, and Code Substitution Delayed).

Table 4. Stepwise logistic regression models of predicting moderate/severe NCD using select Ped-ANAM performance scores*
Ped-ANAM subtestModel 1Model 2Model 3Model 4Model 5 (10)
Slope ± SEPSlope ± SEPSlope ± SEPSlope ± SEPSlope ± SEP
  • *

    NCD = neurocognitive dysfunction; Ped-ANAM = Pediatric Automated Neuropsychological Assessment Metrics; AC = percentage of correct responses per subtest; CVc = coefficient of variation of reaction time for correct responses (consistency or SD of reaction time for a correct response/mean reaction time for correct responses in seconds [MNc]).

AC          
 Continuous Performance−0.04 ± 0.020.043      −0.04 ± 0.020.026
 Spatial Processing−0.17 ± 0.070.020    −0.19 ± 0.150.222  
CVc          
 Continuous Performance  14.42 ± 4.800.003  9.75 ± 6.320.123  
 Matching to Sample  5.21 ± 2.580.043  7.29 ± 4.300.090  
 Mathematical Processing        0.96 ± 2.300.677
 Spatial Processing        6.17 ± 3.790.103
MNc          
 Code Substitution Delayed    −6.73 ± 2.350.004−0.01 ± 0.010.064  
Intercept15.42 ± 6.800.023−11.6 ± 3.08< 0.00013.80 ± 1.930.04815.42 ± 16.640.354−2.40 ± 1.830.188
Figure 1.

Area under (AUC) the receiver operative characteristic curve (ROC). Model 1: ROC curve using stepwise-selected accuracy (AC) measures; Model 2: ROC curve using stepwise-selected coefficient of variation of time required for correct response (CVc) measures; Model 3: ROC curve using stepwise-selected mean time to correct response (MNc) measures; Model 4: ROC curve using stepwise-selected AC, CVc, and MNc measures; Model 5: ROC curve using the model developed in the pilot study (11); Model 6: classification and regression tree analysis (scores defined in Methods). Sens = sensitivity; Spec = specificity.

As an alternative approach, we also explored combinations of Ped-ANAM scores of the above subtests using CART analysis. This resulted in a CART score (range 1–4), with a lower score indicating higher likelihood of moderate/severe NCD. Specifically, if the CVc of Continuous Performance Test was >0.6, the CART score = 1; if the CVc of Continuous Performance Test was ≤0.6 but the CVc of Matching to Sample was >1.4, the CART score = 2; if the CVc of Continuous Performance Test was ≤0.6 and the CVc of Matching to Sample was ≤1.4, but the MNc of Continuous Performance Test was ≤375 milliseconds, the CART score = 3; and if the CVc of Continuous Performance Test was ≤0.6 and the CVc of Matching to Sample was ≤1.4, but the MNc of Continuous Performance Test was >375 milliseconds, the CART score = 4. The CART score of ≥3 had an AUC of 92%, a sensitivity of 83.3%, and specificity of 100% in indicating whether the participant had no NCD versus moderate/severe NCD (Figure 1, Model 6).

Change in speed of sensorimotor processing.

The participants' MNc scores, from performing the Simple Reaction Time subtest in the beginning of the Ped-ANAM session, were subtracted from those when repeating this subtest in the end of the Ped-ANAM session, with larger increases considered to reflect a more pronounced decline in overall speed of sensorimotor processing. Our study showed increases in mean ± SEs of the MNc scores to be 25.6 ± 16.2 msec (P = 0.119), 50.8 ± 22.7 msec (P = 0.028), and 159.6 ± 46.4 msec (P = 0.0003) in the normal cognitive, mild/moderate NCD, and moderate/severe NCD groups, respectively. However, these differences only reached statistical significance when comparing the moderate/severe NCD to the normal cognition group (P = 0.021), while there was only a trend for between the moderate/severe NCD versus the mild/moderate NCD group (P = 0.095).

Responsiveness to change over 18-month time period.

Eighteen-month followup data were available for 24 of the 40 participants with cSLE. In an effort to identify those Ped-ANAM subtests that most closely capture changes in cognition over time, we examined Spearman's correlations between changes in Ped-ANAM scores (AC, MNc, CVc) per subtest and the changes in cognitive performance (Z scores) as per repeat formal neuropsychological testing 18 months apart (Table 5). Differences in the Ped-ANAM scores were more closely associated with changes in visuoconstructional ability, followed by those in processing speed and attention, and to a much lesser degree in working memory.

Table 5. Association of changes of Ped-ANAM scores and ranges of cognitive ability based on formal neuropsychological testing over an 18-month period in 24 cSLE patients*
Ped-ANAM scores/subtestWorking memoryPsychomotor speedAttentionVisuo-constructional ability
  • *

    Values are Spearman's correlation coefficients. Pediatric Automated Neuropsychological Assessment Metrics (Ped-ANAM) change score and changes in Z scores per cognitive domain during formal neuropsychological testing; only values of r > |0.2| are shown and r > |0.4| are statistically significant at P < 0.05. cSLE = childhood-onset systemic lupus erythematosus; AC = percentage of correct responses per subtest; CVc = coefficient of variation of reaction time for correct responses (consistency or SD of reaction time for a correct response/mean reaction time for correct responses in seconds [MNc]).

AC    
 Code Substitution Delayed0.30
 Code Substitution0.25
 Continuous Performance0.340.51
 Logical Relations−0.230.21
 Matching to Sample0.300.280.34
 Matching Grids−0.24
 Mathematical Processing0.51
 Procedural Reaction Time0.390.320.51
 Spatial Processing0.520.200.36
 Sternberg Memory Search0.29
CVc    
 Code Substitution Delayed
 Code Substitution−0.43−0.25
 Continuous Performance−0.28−0.51−0.45
 Logical Relations−0.26−0.31−0.22
 Matching to Sample−0.41−0.27
 Matching Grids
 Mathematical Processing0.24−0.26
 Procedural Reaction Time−0.38−0.49
 Spatial Processing−0.22−0.21
 Sternberg Memory Search−0.25−0.31
MNc    
 Code Substitution Delayed
 Code Substitution
 Continuous Performance0.25−0.23
 Logical Relations
 Matching to Sample−0.40−0.40
 Matching Grids−0.25
 Mathematical Processing
 Procedural Reaction Time
 Spatial Processing−0.24
 Sternberg Memory Search−0.23−0.26
 Simple Reaction Time−0.20

DISCUSSION

Neuropsychiatric SLE is thought to be more common in children as compared to adults, with up to 95% of children manifesting at least 1 symptom of neuropsychiatric SLE, and NCD being present in up to 55% of children according to some studies (2, 27). NCD may be transitory (28) or persistent over years (29) and can occur in patients without other cSLE disease activity (30, 31). Because its symptoms are often subtle, the identification of NCD is sometimes difficult, requiring formal neuropsychological testing to be recognized. In an effort to allow efficient screening for NCD in a clinical setting, we explored the usefulness of the Ped-ANAM (16). In assessing the psychometric properties of the Ped-ANAM in cSLE, we found the administration of this computer test to be feasible, and we also found that selected subtests and indices had good reproducibility, criterion and concurrent validity, and responsiveness to cognitive change, suggesting that these measures may be a useful screening tool for NCD.

Besides newly providing estimates of the reproducibility of the Ped-ANAM, our results support its construct validity for screening overall cognitive function in pediatric populations. Ped-ANAM scores with normal cognition significantly differed from those with the presence of mild/moderate and moderate/severe NCD, as indexed by traditional neuropsychological testing. As would be expected based on the study of the Ped-ANAM in other pediatric diseases, these differences in Ped-ANAM scores were present irrespective of a diagnosis of cSLE. Our results are also in line with previous studies in adults and children with SLE that found a large number of the ANAM and Ped-ANAM scores to be associated with and predictive of formal cognitive test performance (3, 8). Likewise, the responsiveness of the Ped-ANAM to cognitive change is supported by associations of changes in the Ped-ANAM scores with the changes in formal neuropsychological test scores over time.

Low Ped-ANAM TP scores have been reported in numerous clinical populations as demonstration of cognitive difficulties (8, 15, 32–35). However, we did not find the TP scores to be useful in differentiating study participants according to the level of their cognitive function, which is most likely due to atypical performance patterns that led to this variable being an overestimate of actual cognitive abilities. Instead, we found other Ped-ANAM scores that measure accuracy (AC), and the speed of accurate responses in terms of overall mean (MNc) and variability (CVc) were better suited to estimate cognitive function than the TP score (10). Likewise, Hanley and McNeil (24) pointed out the shortcomings of the TP scores on tasks that assess higher cognitive functions (e.g., working memory or executive function).

As noted in our previous study, AC scores for all Ped-ANAM subtests were generally high, irrespective of cognitive abilities of the participants. While this ensures that most individuals can complete the Ped-ANAM successfully (thereby supporting feasibility), the pronounced negative skew of AC scores represents a “ceiling effect,” which limits their utility. It is likely that the ability to identify subtle NCD and possibly changes in NCD over time would be improved if the Ped-ANAM subtests were more difficult, resulting in more normally distributed AC scores. Given the versatility of the Ped-ANAM metrics, this hypothesis could be easily tested by modifying Ped-ANAM task parameters.

We found the Ped-ANAM to have moderate to substantial reproducibility when completed within 1 day, which is similar to what has been reported for the original (adult) ANAM (36). Furthermore, it has been suggested previously that the performance of test takers improves most between the first and second administrations, with marginal improvement upon a third administration (37) on the same day. This raises the possibility that our study provides conservative estimates of the reproducibility of the Ped-ANAM.

Our data confirm the utility of the Continuous Performance Test and Spatial Processing subtests of the Ped-ANAM as screening tasks for NCD (10), and data also indicate that the Matching to Sample and Code Substitution Delayed subtests are similarly useful. For reasons not well understood and different from our previous study, the Mathematical Processing subtest did not add to the identification of NCD. One reason may be that the participants of the current study were younger than those in our earlier study. Alternatively, as the previous study of only 27 cSLE patients was underpowered, the Mathematical Processing subtest association with NCD might constitute a type 2 error. Either way, these inconsistencies suggest that it is premature to use fewer Ped-ANAM subtests in future studies of cSLE-associated NCD.

We also provide initial evidence that the Ped-ANAM is responsive to change in cognition of children and adolescents with cSLE. Hence, even in the context of the ongoing brain development expected in this age group, our findings are in line with those reported for the ANAM in adults with other neuropsychiatric diseases unrelated to SLE (8). Nonetheless, a more detailed evaluation of the ability of the Ped-ANAM to detect clinically relevant changes in cognition of children and adolescents with cSLE remains warranted.

A limitation of our study may be that the sampling strategy used does not allow for the provision of valid estimates of the prevalence of cSLE-associated NCD. This is because we used a convenience sample of patients that was not sampled in a strict consecutive fashion from a clinic. Likewise, the “best friend” matching strategy used to correct for sociodemographic variables affecting cognition resulted in a sizeable number of controls having NCD. None of the above, however, has affected the findings of the reliability and validity of the Ped-ANAM as is supported by exploratory analysis, which assessed the measurement properties of the Ped-ANAM in cases and controls separately (data not shown).

In summary, we established that the Ped-ANAM has promising psychometric properties when used in cSLE and healthy pediatric controls, largely confirming the results of a previous pilot study. This is important because it suggests that this tool, which can be administered in busy clinics by nonspecialist staff, can feasibly track functioning over time for “early warning signs” of cognitive decline, and it can screen for patients who are in need of more specialized neuropsychological followup and interventions. However, prior to the use of the Ped-ANAM in clinical care, a more detailed assessment of the tool's discriminant validity appears warranted, including the delineation of minimum clinical important differences in Ped-ANAM scores. Ongoing studies are expected to provide a solid reference range of Ped-ANAM performance scores in healthy children with different socioeconomic backgrounds.

AUTHOR CONTRIBUTIONS

All authors were involved in drafting the article or revising it critically for important intellectual content, and all authors approved the final version to be submitted for publication. Dr. Brunner had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study conception and design. Brunner, Zelko, Roebuck-Spencer, Beebe, Ying.

Acquisition of data. Brunner, Klein-Gitelman, Zelko, Thomas, Hummel, Nelson, Huggins, Curran, Beebe.

Analysis and interpretation of data. Brunner, Zelko, Roebuck-Spencer, Beebe, Ying.

Acknowledgements

We are indebted to Dr. Joseph Bleiberg, Dr. Dennis Reeves, Kerry Culligan, and Children's National Medical Center for their essential involvement in development of the pediatric version of the ANAM. Furthermore, this study would not have been possible without the dedicated clinical research personnel, namely Aimee Baker, Dina Blair, Adlin Cedeno, and Ashia Ali. We thank Drs. Esi Morgan-DeWitt, Dan Lovell, Alexei Grom, Tracy Ting, Michael Henrickson, and Janalee Taylor, PNP, for providing us with access to their patients with cSLE. We would also like to thank Meredith Amaya, April German, Allison Clarke, Kate Dahl, Antoinette Dezzutti, Lev Gottlieb, Jennifer Heil, Jennifer Keller, Andrew Phillips, Michal Rischall, Rebecca Wasserman Lieb, Lisa Welcome, Donna Diedenhofer, Cindy Scharf, and Mariah Wells for their assistance with neuropsychological testing. A special thanks to Mrs. Elaine Holtkamp for her administrative support of the study and assistance with the manuscript.

Ancillary