- Top of page
- Conflict of Interest
In order to understand the neurobiology of complex behavioral processes like cognitive control (Miller and Cohen 2001), the ability to exert control over one's thoughts and actions, we need to validate high-throughput methods, including unsupervised testing of large numbers of participants in parallel via the Internet. There are numerous benefits to high-throughput behavioral assessment, from achieving sample sizes needed for testing genetic associations, to reducing the logistical hurdles in testing complex familial designs, and refinement of latent behavioral constructs through efficient iterative measurement development. Here, we use this approach to tackle several challenging problems in behavioral research, including efficient examination of a large sample and testing of both parents and offspring, to determine the symptom profile of adolescents in a Web-community sample, and provide initial insights into the heritability of frequently used cognitive tests. Further, we demonstrate the validity of this entirely Web-based design by using traditional construct validity analytic approaches to help overcome lingering skepticism about web assessment.
There are significant gaps in our understanding of not only the neurobiology of cognitive control but even the very definition and expression of the construct. Improved understanding of the component processes attributed to cognitive control through iterative construct and measurement refinement can lead to more tractable studies of the neural and genetic bases of behavior, which in turn may even have clinical implications by helping to elucidate the underlying causes of neuropsychiatric disease. A number of reviews report that working memory and response inhibition are components of cognitive control (Pennington 1997; Sabb et al. 2008). These constructs are also correlated with highly heritable neuropsychiatric diseases including schizophrenia and attention deficit hyperactivity disorder (ADHD), demonstrating that examination of basic psychological processes in healthy community individuals can impact knowledge about major mental illness. Yet, outside of extensive work by Plomin and colleagues on genetic linkage for “g” (e.g., Plomin and Spinath 2002), there are few genetic association studies of cognitive constructs (but see Need et al. 2009). Further, the scant reproducible evidence from psychiatric genetics for categorical disorders (produced in part by noise in the construct definition) should provide an even stronger role for psychological research. The challenge, however, in collecting enough cognitive test data using validated measures to conduct well-powered genetic linkage or association studies remains a barrier.
Using a high-throughput unsupervised platform like the World Wide Web can help to overcome this problem. Although a number of studies have demonstrated strong validity with respect to Web-based testing, broad adoption has continued to elude the field. The web offers virtually limitless sample size, the ability to collect complex family structures in an extremely cost effective manner, and the speed to test and refine constructs and measurements in days or weeks instead of months or years. A number of studies have conducted traditional comparisons of scores for Web- and lab-based cognitive assessment, showing correlations at the ceiling of lab test–retest numbers (e.g., Silverstein et al. 2007; Haworth et al. 2009; Germine et al. 2012). We propose that construct validation procedures are more appropriate for demonstration of the utility and validity of Web-based assessment. Such methods have been used successfully before (Krantz and Dalal 2000; McGraw et al. 2000; Silverstein et al. 2007). Our goal was to build on these previous studies and again specifically highlight the importance of construct development and validation in studying cognitive control via the Web.
Here, we present our Web-based platform to measure cognitive constructs and show strong construct validity using classical test-development tools. We report prevalence of attention symptoms using an adapted scale in our Web-based community cohort, relationships between symptoms and cognitive variables, and suggest heritability of psychological measures. These data begin to build a large normative sample of Web-based responses. We discuss the putative inertial bias in the broad adoption of web testing and suggest how our evidence can help overcome this, toward a path of high-throughput assessment necessary to understand the neurobiology of complex psychological processes.
- Top of page
- Conflict of Interest
Understanding the neurobiology of behavioral constructs like cognitive control will require testing participants using unsupervised and parallel approaches. We present novel findings on symptom prevalence in the web community of adolescents, an interaction between symptoms and cognitive test performance, and strong suggestion of significant heritability of measures frequently used to examine cognitive control. Running hundreds or thousands of participants in lab-based studies is extremely inefficient and practically impossible to execute in a timely manner. Although studies have shown scores on lab-based measures to be highly correlated with those online, there remains skepticism about this approach. In our study, we used typical construct validity tests done for new psychological measures to support our findings. Given this, we suggest consistent use of the Web for cognitive assessment will help overcome continued inertial bias for lab-based cognitive testing and be instrumental in uncovering the genetic bases of behavior.
We sought to characterize a community sample without a diagnosis of ADHD recruited entirely using the web. As such, this is not a “super control” sample (attention symptom sum ranges from 0 to 47). This increases ecological validity and provides additional power for correlations as the data encompasses a large range of scores. It does, however, make it difficult to compare the results directly to prior studies with either clinical patients or typical lab-based control populations, but does represent an important characterization of the symptoms in the community-at-large that can begin to establish Web-normative scores. Our finding of symptom scores across a large range, in children and adolescents without a self-reported diagnosis of ADHD is important and novel for a Web-based community. Recent epidemiological reports from the Centers for Disease Control suggest the community prevalence of a diagnosis of ADHD is over 8% (www.cdc.gov). Few studies, however, have looked broadly at symptoms that exist in the community. Our attention symptom finding supports reports that ADHD-related symptoms are dimensional (Lubke et al. 2009), and should be treated as quantitatively distributed traits in the population. Yet, similar to Lubke et al. (2009), we do find that cognitive test performance changes as function of symptom level, which may suggest different latent classes. These data may improve the ability to track the underlying genetic contribution of these symptoms.
Our findings of high correlations between parent and offspring scores on our cognitive control measures suggest high heritability of these constructs, an important step in investigating genetic associations. Typically, examining heritability is difficult for new computerized measures, as recruiting and testing families in a large enough sample to measure heritability is not feasible. Further, with iterative development of new measures, it becomes more challenging for phenotypes to be adequately validated with respect to genetic studies. Studying a single parent and offspring allows us to compute narrow-sense heritability or what some have called biometric heritability (Lynch and Walsh 1998). These numbers provide a ceiling for additive genetic influences without taking into account shared environment or pure environment factors or epistasis. Our calculations of narrow-sense heritability suggest high heritability but also unsurprisingly that these unmeasured sources of variance do play a role in working memory and response inhibition. They also suggest that some phenotypic indicators may not be useful in genetic association experiments going forward, as they display very low narrow-sense heritability (e.g., Working Memory load accuracy at low loads). These findings suggest our approach is feasible and extremely efficient for examining these questions, but larger pedigree-type data would be ideal for answering these questions. While further research needs to be done to fully address the technical considerations of conducting heritability research remotely using varying equipment, ideally through direct recording of these variables and ensuring family members use different computers, there is evidence in the literature (e.g., Li et al. 2010) suggesting that some of the confounds in computer architectures and peripheral equipment are likely not enough to completely account for our heritability findings. As such, these results may be useful in the future in estimating the size of the effect of hardware/software noise as more detailed data about these sources of noise are studied.
This study also supports our hypothesis about the validity of web assessment of cognitive control. These tests show excellent face validity based on well-established paradigms and demonstrate evidence of construct validity. We also provide additional evidence in showing that the association between both RT and inhibition with the attention symptoms is consistent with the literature (Walshaw et al. 2010). This approach is the same used in other domains of psychological testing (Block et al. 1974; Reynolds and Koback 1995), and while we show somewhat more moderate effect sizes than these psychometrically built instruments, our procedures are identical to other computerized test development. Although typically not seen with new computerized cognitive test development, Gur and colleagues did use a similar approach to demonstrate validity of a larger cognitive test battery (Gur et al. 2010). This is in contrast to previous studies, which have pursued equivalence testing metrics to theoretically ensure tests are identical across testing platforms. Our approach focuses on construct validation using tasks with extremely high face validity. Very few new lab-based variations of cognitive paradigms undergo equivalence testing. Web-based tests that are demonstrated to measure latent constructs of interest should be adequate in assessing cognitive control behavior.
With the ubiquity of the web in our daily lives, it follows that cognitive testing should use web technology, especially as the knee-jerk theoretical biases have been consistently shown to be surmountable. While the sample biases typically associated with Internet-research have been shown to be less problematic in direct examination (Gosling et al. 2004; Haworth et al. 2007), there are typically more demographically varied samples found online, where any study can recruit from millions of potential participants. This is not to suggest that the Web does not have sample biases, but as these studies have shown, the biases are not different from those typically seen in lab-based psychological studies where recruitment is almost never truly random. The benefit with using the Web, is that you can sample from a much larger pool than will be available in a typical lab study (i.e., every demographic category can be found in greater number on the Web than within participation distance of any single institution).
The primary concern about web testing, however, has been response bias. There is a large body of evidence showing high correlations (>0.7–0.8) between web and lab assessment in the same individuals (Buchanan and Smith 1999; Krantz and Dalal 2000; Gosling et al. 2004; Bedwell and Donnelly 2005; Haworth et al. 2007; Silverstein et al. 2007; Younes et al. 2007; Germine et al. 2012). Buchanan (2003) argues that solely because an assessment was adapted from lab- to Web-based format one cannot assume the newer version has the same psychometric properties. While true, to conclude that this means that web versions are not useful is premature, rather a web test should be considered a new measure, with its own psychometric properties and norms. The construct validation approach used here and by others previously (Krantz and Dalal 2000; McGraw et al. 2000; Silverstein et al. 2007) builds upon the growing evidence base for the valid adoption of Web-based assessment of cognitive control.
Web testing provides novel experimental design opportunities for examining the underlying genetic substrates of behavior. While methodologies for examining the genetic associations have improved dramatically in the last several years, efforts aimed at clarifying phenotypic expression have lagged, especially in neuropsychiatry (Sabb et al. 2009). The common misconceptions about the pitfalls of web testing have shown to be no worse than pitfalls seen in laboratory testing, but efficiency and cost-effectiveness are unparalleled with web testing. We demonstrated the power of this approach by efficiently recruiting a large family sample, which revealed the prevalence of subclinical attention symptoms in the web community. We also demonstrated an interaction between cognitive test performance and symptom level that may have implications for more broadly understanding ADHD. Finally, our data suggests a “ceiling” for the heritability of these Web-based cognitive control measures. This may aid cognitive control phenotype selection for genetic analyses going forward, as several indicators had particularly low ceilings and could be avoided. More broad adoption of the web for testing is needed to demonstrate test–retest reliability in web scores and establishment of Web-based norms. If successful, this approach could greatly increase our ability to understand the underlying neurobiology of behavioral constructs that are core components of neuropsychiatric diseases.