The validation of a new online cognitive assessment tool: The MyCognition Quotient

Abstract Objectives Cognitive impairment affects many psychiatric patients, influences daily functioning, and should be an important treatment focus. Assessment of cognitive status is crucial in cognitive remediation studies. However, current test batteries have limitations. A new, online tool, the MyCognition Quotient (MyCQ), was developed to assess cognition within 30 min. We present the psychometric properties and aim to determine the validity of the MyCQ by comparing it with the Cambridge Neuropsychological Automated Test Battery (CANTAB). Methods Eighty‐seven patients diagnosed with obsessive compulsive disorder, schizophrenia/schizoaffective disorder, or major depressive disorder were included. Patients completed both the MyCQ and CANTAB. Results Our hypothesized domains of psychomotor speed, attention, episodic memory, working memory, and executive functioning were confirmed by principal component analysis. The MyCQ total score correlated highly with the CANTAB total score. The MyCQ domains of psychomotor speed, attention, and episodic memory showed moderate to high correlations with corresponding CANTAB domains. Working memory and executive functioning had limited divergent validity. Conclusion The MyCQ appears to be a promising instrument for assessing cognition online within a mixed psychiatric population. It is cost‐efficient, easily administered, and usable in different psychiatric populations, which makes it a good candidate for both clinical and community studies.

show promising results for patients with a psychotic disorder (Anaya et al., 2012;Grynszpan et al., 2011;Wykes, Huddy, Cellard, McGurk, & Czobor, 2011) as well as patients with other psychiatric diagnoses (Motter et al., 2016;Tchanturia, Lounes, & Holttum, 2014). To further develop and validate these cognitive remediation programs and for successful implementation in the (outpatient) clinic, assessment of the patient's cognitive status is essential. Unfortunately, although many good instruments are available for assessing cognitive functioning, their content, methods, and applications vary widely (Keefe et al., 2004). They require specialist supervision making them timeconsuming and expensive and are usually confined to a clinical setting.
A new, web-based tool, the MyCognition Quotient (MyCQ), has been developed to overcome these drawbacks. The aim of this paper is to present the MyCQ and report its psychometric properties.
To measure the full spectrum of cognitive functioning, traditionally, a broad range of separate tests are combined into a neurocognitive test battery. There is an extensive choice of tests available that have proven themselves valid, reliable, and sensitive over the last decades in measuring specific aspects of cognitive functioning.
However, the combination of a sufficient number of these traditional tests into a complete test battery can lead to a lengthy assessment, taking up to several hours to complete. This can be burdensome for a patient and affect their motivation and performance, which might even lead to trial dropout. Furthermore, leaving researchers free to create their own test battery can result in a wide variety of test combinations. This negatively influences consistency and generalizability of the results, and thus hinders the standardized evaluation of new interventions aiming to improve cognition. This method also requires test supervisors to be proficient in many different tests. In the last decade, this issue has fortunately received more attention, and a number of guidelines and standard batteries for measuring cognition in psychiatric patients were developed. The Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS) Consensus Cognitive Battery for the assessment of cognitive functioning in schizophrenia (Nuechterlein et al., 2008) could now be considered the gold standard for cognitive assessment in schizophrenia, being recommended by the Food and Drug Administration for clinical trials of cognitive enhancement therapies in schizophrenia. The Brief Assessment of Cognition (BACS; Keefe et al., 2004) is another standardized comprehensive battery for schizophrenia, of which a new version is currently available for affective disorders (BAC-A).
Nowadays, ease of access of test instruments is important for both research as well as clinical practice. Initially, cognitive assessment required the minimum involvement of pen and paper, but often also utilized various props such as colorful blocks (Wechsler, 2008) or plastic balls (Culbertson & Zillmer, 1998 (Gualtieri & Johnson, 2006). Furthermore, the BACS was recently transformed into a digitalized version in the form of an app that can be used on an iPad (Atkins et al., 2017). These computerized batteries have the advantages of having standardized and consistent administration and automated response recording and scoring, reducing errors in scoring. However, these computerized tests are still dependent on software and hardware that can be relatively expensive, are often tied to a specific location, and still require supervision by a trained expert. Web-based and self-administered assessment would increase accessibility and reduce costs, which could be advantageous for both research as well as clinical practice. In the last years, the mobile, self-administered assessment of cognition has slowly started to make ground, but not many validation studies have yet been done. The THINC-integrated tool (THINC-it) cognitive screener app (McIntyre et al., 2017) has recently been validated for subjects with depression.
In the current paper, we study an online and patient-administered instrument for patients with varying psychiatric disorders, the MyCQ.
The MyCQ was developed to assess the broad cognitive status of patients in a relatively quick and easy manner. The instrument was developed with the primary care setting in mind. It is intended as a selfadministered tool, to cut down on clinical workload and open up cognitive assessment to larger numbers of patients. Originally, the MyCQ was intended to be used in unison with an online cognitive training application, where it tracks progress and helps determine which cognitive domains require most training. It furthermore could be used for research purposes as well as in care settings, fulfilling a screening function or to help keep track of cognitive status longitudinally. The MyCQ aims to assess five cognitive domains through 10 short subtests and can be completed online via PC or iPad. Furthermore, the MyCQ was developed to be used transdiagnostically, without a specific patient population in mind.
In the present study, we assessed the psychometric properties of the MyCQ in a transdiagnostsic sample. Internal consistency and structure of the instrument were evaluated. Convergent and divergent validity was determined by comparing the MyCQ with the CANTAB.
Finally, we examined the ability of the cognitive domains to relate to age and premorbid IQ. Axis I disorder of schizophrenia/schizoaffective disorder, obsessive compulsive disorder or major depressive disorder. Exclusion criteria were a high risk of suicide, an unstable comorbid medical disorder, meeting the criteria for a substance use disorder within the last 3 months, a history of a clinically significant abnormality of the neurological system or seizure, and a premorbid IQ below 70. All subjects received a compensation of €40 and a travel cost reimbursement.
Written informed consent was obtained from all subjects and from the legal guardians when the subject was younger than 18 years.

| The MycQ
The MyCQ was developed to assess cognitive functioning in a fast and efficient way. It is available online and was developed to be selfadministered in a variety of settings. The MyCQ consists of 10 subtests that are hypothesized to measure five primary cognitive domains: psychomotor speed, attention, episodic memory, working memory, and executive functioning. Although researchers do not fully agree on how cognitive functioning can be divided into different domains, the MATRICS initiative has identified seven separate cognitive domains (speed of processing, attention/vigilance, working memory, verbal learning and memory, visual learning and memory, reasoning and problem solving, and social cognition) that are often affected in schizophrenia and that are now leading in studies about cognitive function in schizophrenia (Nuechterlein et al., 2004). The MATRICS domains are similar to those of the MyCQ. Although the MyCQ considers episodic memory as one domain, the battery includes a verbal as well as a visual memory task. The MyCQ does not include a social cognition domain, because it focuses specifically on neurocognitive functioning. Social cognition is recognized as an important construct that is often impaired in schizophrenia (Green & Leitman, 2008) as well as in mood disorders (Ladegaard, Larsen, Videbech, & Lysaker, 2014) and is associated with functional impairment (Fett et al., 2011). However, social cognition and nonsocial neurocognitive functioning have been shown to be related but distinct phenomena (Fett et al., 2011;Hasson-Ohayon, Goldzweig, Lavi-Rotenberg, Luther, & Lysaker, 2018;van Hooren et al., 2008). With the five domains assessed by the MyCQ, the primary neurocognitive functions are covered. The MyCQ uses renowned paradigms that index the key areas of cognitive function. Each subtest is based on a reliable and well validated paperand-pencil cognitive test. In the selection of the subtests, it was important that subtests were relatively brief and would be easy to use without expert administration. A detailed description of the subtests and their paper-and-pencil equivalents and the corresponding hypothesized domains are presented in Table 1 and Data S1. Written instructions with illustrative pictograms are included in the instrument and are provided prior to each subtest being taken. After the instruction, a short practice session is also provided. Subjects receive feedback on their performance after this practice in the form of the number of errors made and reaction speed and then proceed with the actual test.
For every subtest, two outcome variables are recorded: mean latency and total number of errors, resulting in a total of 20 outcome variables.

| Cambridge Neuropsychological Test Automated Battery
To assess the convergent validity of the MyCQ, the CANTAB (Cambridge Cognition Ltd., 2008) was employed as second measurement of cognitive functioning. This computerized test battery has been extensively used in clinical practice as well as in scientific studies in a wide range of disorders and healthy controls. It is mentioned in over 2,000 scientific publications, and its ability to adequately discriminate between healthy adults and various (neuro) psychiatric populations has been confirmed (Egerhazi, Berecz, Bartok, & Degrell, 2007;Haring, Mottus, Koch, Trei, & Maron, 2015). It shows moderate correlations with traditional neuropsychological assessments (Smith, Need, Cirulli, Chiba-Falek, & Attix, 2013). The CANTAB includes a variety of subtests that are delivered on a touchscreen computer and can be combined into different test batteries. We used the five CANTAB subtests choice reaction time (CRT), rapid visual information Processing (RVP), verbal recognition memory (VRM), spatial working memory (SWM), and intra-extra dimensional set shift (IED) to measure the five domains of psychomotor speed, attention, episodic memory, working memory, and executive functioning.

| Premorbid IQ
Premorbid intelligence was assessed by using the Dutch version of the National Adult Reading Test (NART; Bright, Jaldow, & Kopelman, 2002), which is a valid estimate of a person's premorbid level of intellectual ability. The NART is an untimed measure, consisting of 50 words with atypical phonemic pronunciation. Subjects are presented these words on a list and are asked to read each aloud.

| Procedure
Subjects were assessed as part of a larger randomized controlled trial testing the effectiveness of a new cognitive remediation game, AquaSnap™. When approached for participation, subjects received written and oral information about the study goals and procedures.
After signing informed consent and being negatively screened for the exclusion criteria, subjects were interviewed regarding their clinical, social, and cognitive status. Then, the cognitive assessment began with the NART. Thereafter, the subjects completed the CANTAB subtests followed by the MyCQ. The MyCQ was completed on a laptop with a mouse. A trained psychology student was present during the full assessment to introduce and supervise tasks and to make sure participants understood the instructions. Total testing time for each subject could take up to 4 hr. Because some subjects experienced difficulty sustaining their attention over such a period of time, sometimes two appointments were necessary to complete the measurement. The second appointment was always planned as soon as possible following the first appointment. Subjects could ask for a short break when needed during the testing session.

| Statistical analyses
The inversed, so that a higher score meant more impaired cognition. The decision about which outcomes to combine into a composite score was based on factor structure for the MyCQ. For the CANTAB, we chose one subtest per cognitive domain and combined the key variables (CRT mean correct latency, RVPA and RVP correct rejections, VRM free recall (short term) and recognition (short and long term), IED total errors, and spatial working memory total errors) into a composite score.
We examined the sensitivity of the MyCQ to differentiate between in-group differences by examining Pearson correlations between cognitive domain scores and sum scores and NART and age.
All statistical analyses were performed in SPSS (version 22) for Windows, and statistical significance was set at the 0.05 level.

| RESULTS
A total of 87 subjects were included in this study (44 male). Age ranged from 16 to 56 years (M = 32.0, SD = 10.44). The mean total duration of the MyCQ was 30.8 min (range = 24.7-64.8, SD = 6.43 min). Table 2 presents the sample's demographic characteristics. Before further analyses, the data were inspected for outliers. Nine MyCQ and five CANTAB data points were identified as outliers for falling more than four standard deviations from the mean. These severely deviating data-points most likely occurred due to measurement error (e.g., misplacement of the fingers on the keyboard or starting the subtest before instructions were clear) and were excluded from further analy-

| Convergent and divergent validity
One subject was missing CANTAB data, this subject was excluded from further analyses. The MyCQ composites based on the PCA were compared with CANTAB composites using Pearson correlations. The results are presented in Table 4. Significant correlations were found between every matched domain, with the highest correlations between psychomotor speed (p < 0.001) and episodic memory (p < 0.001). All z-scores were also summed to form one MyCQ or CANTAB sum score and were highly correlated (N = 77, r = 0.650, p < 0.001) as can be seen in Figure 1.
For psychomotor speed and attention, divergent validity appeared to be reasonable, with lower correlations between nonmatching domains compared with the matching domains of CANTAB and MyCQ.
Although the MyCQ domains of episodic memory, working memory, and executive functioning showed significant correlations with their corresponding CANTAB domains, these components showed comparable or stronger relationships with nonmatching CANTAB domains.

| Associations between MyCQ scores and subject characteristics
Pearson correlations were calculated between MyCQ scores and age and NART with a significance level of 0.05. Higher age related to slower psychomotor speed (r = 0.224, p = 0.037) and worse performance on the working memory (r = 0.232, p = 0.033) and executive functioning (r = 0.393, p < 0.001). A higher NART score was associated with better MyCQ performance in psychomotor speed (r = −0.217, p = 0.044) and executive functioning (r = −0.254, p = 0.019).
In contrast, when comparing the CANTAB domain scores with these subject characteristics, only two significant correlations were found. The CANTAB sum score (r = 0.229, p = 0.038) and psychomotor speed domain (r = 0.223, p = 0.040) were significantly related to age, with higher age relating to worse performance. There were no associations between NART score and CANTAB domain or sum score.

| DISCUSSION
The results of the current study provide initial evidence that the  working memory, verbal memory and visual memory, reasoning and problem solving, and social cognition (Nuechterlein et al., 2004).   be dropped from further analyses. Four more outcome measures were excluded due to either not loading highly on any of the retained factors or loading similarly high on more than one factor. Removing outcome measures that complicate the factor structure of an instrument is common practice. Simple structure, in which each item loads relatively strong on one factor and small on others, helps to achieve easy and meaningful interpretation (Thurstone, 1947). All subtests are still incorporated in the final MyCQ total score and subdomains, providing evidence that the five cognitive domains are covered by the 10 subtests of the MyCQ. More than 75% of the total variance observed was explained, a good result compared with other studies (Henson & Roberts, 2006). Nevertheless, our sample was quite small for this type of analysis, which could be of influence on the accuracy and generalizability of the model. Although opinions on minimum sample sizes for PCA differ, most rules of thumb ask for at least 10:1 ratio of N to variables (Osborne & Costello, 2004). Until the current results are replicated in a larger study population, caution in the interpretation of these domains is warranted.
Most MyCQ domains are moderately associated with each other.
This finding is not surprising and has been noted by scientists since the beginning of the 20th century when Charles Spearmen first proposed the existence of a general intelligence factor, or g factor (Spearman, 1904 although previous studies suggest that these are overlapping concepts (Chan, Shum, Toulopoulou, & Chen, 2008;McCabe, Roediger, McDaniel, Balota, & Hambrick, 2010), and the MyCQ also shows a moderate association between the two. Finally, the errors on the  (Lenehan, Summers, Saunders, Summers, & Vickers, 2016). This highlights the overall difficulty of measuring human cognition. The poor divergent validity of the domains of working memory and executive functioning warrants some caution when interpreting the results of the MyCQ.
Although using the MyCQ for assessing global cognitive functioning by using the overall score seems valid, for the confirmation of impairments in the separate domains-especially those of working memory and executive function-additional tests might be considered.
The MyCQ is somewhat sensitive to differences in age and premorbid IQ. Better MyCQ psychomotor speed and executive functioning scores are associated with a lower age and a higher premorbid IQ. Working memory also declines with age. In contrast, the CANTAB domains are not associated with premorbid IQ and only the sum score and psychomotor speed show age-related decline. When using the Bonferroni correction for multiple comparisons, only the association between age and executive functioning retains its significance. However, the necessity to Bonferroni correct has been disputed. Especially with a relatively small sample size such as in our study, the probability of Type II errors greatly increases with this method, while the power to find small effects is reduced (Nakagawa, 2004 Despite these limitations, the advantages of assessing cognition with an online assessment tool such as the MyCQ are numerous.
Computerized tests make consistent and precise administration and scoring possible, reducing measurement error and examiner bias.
Computerized testing has been used before, but offering it through an online portal is a new development and could be the next step in neurocognitive assessment. One of the important advantages of self-administered and online assessment, over and above the costefficiency, is that it enables large scale assessment in a range of settings, both in the clinic and the community. This could be especially important for prevention studies and interventions. There is evidence that cognitive impairment can precede other clinical symptoms. This is especially true for psychotic disorders (Fuller et al., 2002), but might also be the case for mood disorders although results are still inconclusive (Allott, Fisher, Amminger, Goodall, & Hetrick, 2016).
Neurocognitive functioning might well be a shared risk factor for overall mental health and global functioning. Although more studies are necessary to enlighten how cognition influences later functioning, early detection of impairment in neurocognitive functioning might be important and could be widely implemented with an online instrument. However, the usability of the MyCQ for psychiatric patients in settings outside the clinic should be further established. The MyCQ already has been used to assess large school populations of 600-800 students, confirming its usability in community studies for healthy populations (Ratto, Cliveden, & Sparrowhawk, 2017).
The MyCQ opens up the possibility of home assessment, which might especially be cost and time efficient and less burdensome for patients. The current paper did not investigate assessment at home, and it is still unclear how home assessment might affect the validity and reliability of the results. Uncontrolled confounders, such as distractions, assistance, or substance use, might interfere with reliable assessment. However, in some situations, home assessment might even provide an ecologically more valid estimation of a patient's cognitive functioning in their daily life. The effects on the reliability and validity of the MyCQ when assessing cognition at home should be further investigated. A number of subjects in this study also completed a follow-up measurement of the MyCQ at home without problems, providing first support for the usability of the instrument for home assessment.
In summary, the MyCQ appears to be a promising instrument for assessing cognitive functioning online within a mixed psychiatric population. It compares reasonably well with the CANTAB, and its short duration and self-administration capability potentially reduces health care costs. More studies with different populations, for example, Parkinson's disease (van de Weijer et al., 2016) and breast cancer patients with depression, and different neuropsychological assessment batteries are in progress to further validate the MyCQ and investigate its usability within other populations.