A systematic review of the diagnostic accuracy of automated tests for cognitive impairment

Objective The aim of this review is to determine whether automated computerised tests accurately identify patients with progressive cognitive impairment and, if so, to investigate their role in monitoring disease progression and/or response to treatment. Methods Six electronic databases (Medline, Embase, Cochrane, Institute for Scientific Information, PsycINFO, and ProQuest) were searched from January 2005 to August 2015 to identify papers for inclusion. Studies assessing the diagnostic accuracy of automated computerised tests for mild cognitive impairment (MCI) and early dementia against a reference standard were included. Where possible, sensitivity, specificity, positive predictive value, negative predictive value, and likelihood ratios were calculated. The Quality Assessment of Diagnostic Accuracy Studies tool was used to assess risk of bias. Results Sixteen studies assessing 11 diagnostic tools for MCI and early dementia were included. No studies were eligible for inclusion in the review of tools for monitoring progressive disease and response to treatment. The overall quality of the studies was good. However, the wide range of tests assessed and the non‐standardised reporting of diagnostic accuracy outcomes meant that statistical analysis was not possible. Conclusion Some tests have shown promising results for identifying MCI and early dementia. However, concerns over small sample sizes, lack of replicability of studies, and lack of evidence available make it difficult to make recommendations on the clinical use of the computerised tests for diagnosing, monitoring progression, and treatment response for MCI and early dementia. Research is required to establish stable cut‐off points for automated computerised tests used to diagnose patients with MCI or early dementia.

These estimates also vary significantly depending on the definitions used in different studies. For example, a large population-based study of older-aged individuals in the United Kingdom 7 reported prevalence estimates of individuals not classified from current MCI definitions were variable (range, 2.5-41.0%). In addition, the rates of progression from MCI to dementia varied from 3.7% to 30.0%. 7 Evidence from neuropathological and neuroimaging studies suggests that biological changes associated with dementia occur long before the onset of symptoms. 8 This has given rise to the concept of mild cognitive impairment (MCI), which is the state between the cognitive changes of normal ageing and early dementia. [9][10][11] Mild cognitive impairment refers to the clinical condition used to describe people whose cognitive function is below that of the normal population for their educational level and age but who do not have any loss of functional abilities or skills. [11][12][13][14] It is a heterogeneous state, with possible trajectories including Alzheimer disease (AD), Lewy body dementias, and even reversion to normal cognitive functioning. 15 The difference between MCI and early dementia is based on the level of cognitive decline and pattern of change in mood and behaviour. Individuals diagnosed with early dementia present with multiple cognitive deficits, and their memory loss is sufficient to impact everyday social and occupational functioning. Among the 4 most common medical conditions causing dementia are AD, vascular conditions, frontotemporal atrophy, and Lewy body disease. Irrespective of the primary reason, the cognitive prognosis for people with most types of dementia is usually poor. 16,17 There are a number of pen-and-paper-based tools as suitable tests for screening people for cognitive impairment, for example, the General Practitioner Assessment of Cognition, 6-item Cognitive Impairment Test, and Mini-cog assessment instrument. 18,19 There are different pen-and-paper tests used to aid diagnosis by specialists for MCI and early dementia, for example, the Dementia Toolkit for Effective Communication, 20 Montreal Cognitive Assessment, 21 and Saint Louis University Mental Status. 22 However, these specialist tests can be expensive and time-consuming. 23 More recently, several automated tests have been developed, 24,25 which may be uniquely suited to early detection of changes in cognition, by, for example, covering a wider range of ability to precisely record accuracy and speed of response with a level of sensitivity not possible in standard administrations. 23 The rationale for this review is to determine whether automated computerised tests for cognitive impairment have the potential to contribute to early diagnosis and simplify the current method of monitoring progression and treatment response compared with standard clinical practice.

| METHODS
A systematic review was performed to describe the diagnostic accuracy of automated tests to detect MCI and early dementia as well as investigate their role in monitoring disease progression and response to treatment. The methodology and reporting of this review followed the guidance set out by the Cochrane Handbook for Diagnostic Test Accuracy Reviews. 26 See Appendix S1 found in the Supporting Information for an abbreviation list.

| Criteria for considering studies for this review
Any study assessing the diagnostic accuracy of automated computerised tests to diagnose or monitor MCI or early dementia against a reference standard was considered for inclusion. Case studies and qualitative studies were excluded. Studies or diagnostic tools published in a non-English language were also excluded.

| Participants
Participants were people with MCI or early dementia diagnosed by any recognised diagnostic standard.

| Index tests
The index tests considered for inclusion were automated computerised tests of cognitive impairment, which can either be selfadministered or interviewer administered.

| Reference standard
The reference standard for this review is the clinical diagnosis of MCI and early dementia using a diagnostic criteria, for example, the International Classification of Diseases 2 edition 10 and the Diagnostic and Statistical Manual of Mental Disorders editions 4 and 5 (DSM-IV and DSM-V, respectively). 27 It is recognised that clinical diagnosis itself has a degree of variability, but this is not unique to dementia studies and does not invalidate the basic diagnostic test accuracy approach. • A number of automated tests for diagnosing and monitoring progression of cognitive impairment have been developed, which need to be used in conjunction with clinical assessment.

| Search methods for identification of studies
• The overall quality and quantity of the available evidence are insufficient to make recommendations on the clinical use of these automated computerised tests.
• Further research is required to examine the cut-off points for different populations in automated tests for diagnosing and monitoring progression and treatment response of MCI and early dementia. different databases is provided in Appendix S3 found in the Supporting Information, and were managed in Endnote X7.

| Selection of studies
Two reviewers independently screened all relevant titles and abstracts and full-text articles for inclusion. Any disagreements were resolved by discussion with a third reviewer.

| Data extraction and management
Data extraction forms were developed and piloted in an Excel spreadsheet by using 2 of the included studies. Data on study design, population characteristics, and outcomes were extracted by one reviewer and independently checked for accuracy by a second reviewer, with disagreements resolved through discussion with a third reviewer when necessary. The extracted data included information on the reference standard, index test, cut-off points, and the measures of diagnostic test accuracy including sensitivity, specificity, receiver operating characteristic curve, and the area under the curve (AUC) for discriminating amongst MCI, early dementia, and cognitively healthy individuals.

| Assessment of methodological quality
The methodological quality of the included studies was assessed by one reviewer and independently checked for accuracy by a second reviewer using the Quality Assessment of Diagnostic Accuracy Studies tool, 28 which is recommended by the Cochrane Diagnostic Test Accuracy Reviews Guidelines. 29 This tool is designed to evaluate the risk of bias and applicability of primary diagnostic accuracy studies using signalling questions in 4 domains: patient selection, index test, reference standard, and flow and timing. Out of authors of all the included studies approached with a request for specific sensitivity and specificity data, only 2 provided these data.

| Statistical analysis and data synthesis
It was not possible to perform a meta-analysis because of noncomparable data; the study designs varied, the cut-off points for the primary outcome measure were heterogeneous, and the summary statistics were often inconsistently reported. A narrative synthesis of the results of the included studies was conducted.

| Patient and public involvement
An advisory group comprising clinicians and service users guided the team during the review. A call for participation was sent through frontline groups, for example, Alzheimer's Society and Dementia UK, to identify people interested in giving feedback on the results of the review and on the final report. The review team took guidance from these agencies and INVOLVE 30 for planning and facilitating the meetings. In addition to the 16 included studies, 4 trials were identified during hand searching (Appendix S4 found in the Supporting Information).
The authors of these studies were approached by email and telephone for results, but no responses were received. The summary of the included 16 studies is presented in Table 1; there were 7 cohort studies, 7 case-control studies, and 2 cross-sectional studies. 40

| Findings
The diagnostic accuracy of 11 automated computerised tests for the detection of MCI and/or early dementia without co-morbidities was evaluated in 15 studies and 1 study with co-morbidity. 43 The details of the index tests are summarised in Table 2. Pooling of data from these 16 studies was considered inappropriate since there were few studies evaluating the same index test in the same population, and it was only possible to extract 2 × 2 data from 5 of the 16 studies.

| Studies reporting on diagnostic accuracy outcomes with a 2 × 2 table
There were 5 studies that reported diagnostic accuracy outcomes in a 2 × 2 table as described in Table 3. Two studies reported the diagnostic accuracy outcomes for MCI, 3 studies reported outcomes for early dementia, and 1 study reported combined outcomes for both MCI and early dementia.

| Mild cognitive impairment
Juncos-Rabadan et al 35   and high specificity (100%) and an AUC of moderate discrimination (0.780) between the early-dementia group and non-early-dementia group.

Mundt et al 41 assessed the Computer Automated Telephone
System and reported moderate sensitivity (79.17%) and high specificity (83.8%) for this test.

| MCI/early dementia
One study evaluated CANTAB-PAL. The authors reported high sensitivity (96.9%) and high specificity (80.8%) with an AUC of good discrimination (0.897) between the MCI/early-dementia group and non-MCI/early-dementia group.     3.3 | Studies reporting on diagnostic accuracy outcomes without a 2 × 2 table The authors of 11 studies reported diagnostic accuracy outcomes for 9 different index tests without using 2 × 2 data as tabulated in Table 4.
Instead, they calculated optimal sensitivity and specificity values using receiver operating characteristic curve analysis.

| Mild cognitive impairment
Eight studies reported the diagnostic accuracy outcomes for MCI.  Mindstreams with an overall AUC of 0.886, which showed a good ability to discriminate between the early-dementia group and the nonearly-dementia group.

| HIV-associated neurocognitive disorders
One study 43

| Methodological quality
The methodological quality of the included studies was assessed using the Quality Assessment of Diagnostic Accuracy Studies tool as summarised in Figure 2.
The risk-of-bias criterion for patient selection was high for 7 studies because a case-control study design had not been avoided (see Appendix S6 found in the Supporting Information). Seven studies were judged to be at unclear risk in the index test criteria for risk of bias since the threshold values for the index tests were not prespecified. There was high concern regarding the applicability of the index test for all of the studies because the interpretation of the index test was different from the review question, since it is not possible to establish diagnosis of MCI and early dementia using automated computerised tests in isolation; specialist expertise is necessary to establish a diagnosis.
The reference standard domain for the risk of bias was unclear in 8 studies since it was not possible to ascertain whether reference standard results were interpreted without knowledge of the results of the index tests. All but one study 38 were judged to have low concern for applicability regarding the reference standard since it used a consensus of 2 clinicians' opinions as the reference standard. In the flow and timing domain for the risk of bias, a judgement of unclear risk of bias was given to 2 studies 35,43 since attrition or timing was not described in the papers.
However, 14 studies were assessed as being at low risk because all patients had received the same reference standard and all patients were included in the analysis. There was a high concern in the domains of applicability for 16 studies. Of the 16 studies, only 1 was judged to be at low of risk of bias across the 4 domains examined 39 ; despite this, the overall quality of the included studies was considered to be good.

| Patient and public involvement
Data from the included studies were presented and discussed with a service user. The structure of the meeting is described in Appendix S5 found in the Supporting Information. The service user thought that all of the index text domains needed to be tested to enable a comprehensive overview of any suspected cognitive impairment. His view was that more information on key domains would help clinicians and patients address the challenges faced by patients with MCI or early dementia.
The service user raised concerns about the age of the study participants since there were no tests that assessed cognitive impairment in people over the age of 90 years. Another concern was the effect of little or no education on the ability to perform well on the test. The importance of the index tests being user-friendly and acceptable to patients was also highlighted. He also stated a preference for desktop computers over touch screen test, in case a patient had tremors. He also highlighted the importance of ensuring that the colour palette in visual components of the tests had a sharp contrast because it is likely that older people will have problems with their eyesight. He also stated that some people might become frustrated with tests that lasted longer than 40 minutes.

| DISCUSSION
In assessing the diagnostic accuracy of a test, an index test with high specificity is preferable for diagnosis, and high sensitivity is preferred None of the previously conducted relevant reviews in this area conducted a diagnostic accuracy review. 23,48,49 They were narrative reviews that provided a summary of the battery of tests used and rated this evidence on validity and reliability, comprehensiveness, and usability. This review focused on computerised tests that were self-administered and had a minimum level of involvement from professionals. In line with the findings of this review, the authors of the other reviews concluded that there is significant difference in automated computerised tests, and hence, they must be judged on a case-by-case basis. 23 More research is required to establish stable cut-off points for each automated test used to diagnose patients with MCI or early dementia. An important consideration is testing the cut-off points in specific patient populations, for example, in patients of different age groups or education levels and from different geographical regions.
Another area for future research is providing more information on

| Strengths of this review
The search strategy for this review was extensive. The methodological rigour of the review process was enhanced by the use of 2 assessors to perform citation screening, quality assessment, and data extraction/ checking. All of the primary study authors were contacted and asked to fill in the contingency tables. A patient and public involvement exercise was also conducted.